-- Logs begin at Mon 2022-08-29 20:49:40 UTC, end at Mon 2022-08-29 21:54:40 UTC. -- Aug 29 21:19:10.251680 dv-cinder-854267-3-iscsi systemd[1]: Starting Devstack devstack@keystone.service... Aug 29 21:19:10.260832 dv-cinder-854267-3-iscsi devstack@keystone.service[81425]: [uWSGI] getting INI configuration from /etc/keystone/keystone-uwsgi-public.ini Aug 29 21:19:10.262845 dv-cinder-854267-3-iscsi devstack@keystone.service[81425]: *** Starting uWSGI 2.0.18-debian (64bit) on [Mon Aug 29 21:19:10 2022] *** Aug 29 21:19:10.262845 dv-cinder-854267-3-iscsi devstack@keystone.service[81425]: compiled with version: 10.0.1 20200405 (experimental) [master revision 0be9efad938:fcb98e4978a:705510a708d3642c9c962beb663c476167e4e8a4] on 11 April 2020 11:15:55 Aug 29 21:19:10.262845 dv-cinder-854267-3-iscsi devstack@keystone.service[81425]: os: Linux-5.4.0-72-generic #80-Ubuntu SMP Mon Apr 12 17:35:00 UTC 2021 Aug 29 21:19:10.262845 dv-cinder-854267-3-iscsi devstack@keystone.service[81425]: nodename: dv-cinder-854267-3-iscsi Aug 29 21:19:10.262845 dv-cinder-854267-3-iscsi devstack@keystone.service[81425]: machine: x86_64 Aug 29 21:19:10.262845 dv-cinder-854267-3-iscsi devstack@keystone.service[81425]: clock source: unix Aug 29 21:19:10.262845 dv-cinder-854267-3-iscsi devstack@keystone.service[81425]: pcre jit disabled Aug 29 21:19:10.262845 dv-cinder-854267-3-iscsi devstack@keystone.service[81425]: detected number of CPU cores: 4 Aug 29 21:19:10.262845 dv-cinder-854267-3-iscsi devstack@keystone.service[81425]: current working directory: / Aug 29 21:19:10.262845 dv-cinder-854267-3-iscsi devstack@keystone.service[81425]: detected binary path: /usr/bin/uwsgi-core Aug 29 21:19:10.262845 dv-cinder-854267-3-iscsi devstack@keystone.service[81425]: your processes number limit is 47680 Aug 29 21:19:10.262845 dv-cinder-854267-3-iscsi devstack@keystone.service[81425]: your memory page size is 4096 bytes Aug 29 21:19:10.263481 dv-cinder-854267-3-iscsi devstack@keystone.service[81425]: detected max file descriptor number: 2048 Aug 29 21:19:10.263481 dv-cinder-854267-3-iscsi devstack@keystone.service[81425]: lock engine: pthread robust mutexes Aug 29 21:19:10.263481 dv-cinder-854267-3-iscsi devstack@keystone.service[81425]: thunder lock: enabled Aug 29 21:19:10.263481 dv-cinder-854267-3-iscsi devstack@keystone.service[81425]: uwsgi socket 0 bound to UNIX address /var/run/uwsgi/keystone-wsgi-public.socket fd 4 Aug 29 21:19:10.263481 dv-cinder-854267-3-iscsi devstack@keystone.service[81425]: Python version: 3.8.10 (default, Jun 22 2022, 20:18:18) [GCC 9.4.0] Aug 29 21:19:10.302853 dv-cinder-854267-3-iscsi devstack@keystone.service[81425]: Python main interpreter initialized at 0x55d33ffeb9c0 Aug 29 21:19:10.302853 dv-cinder-854267-3-iscsi devstack@keystone.service[81425]: python threads support enabled Aug 29 21:19:10.302853 dv-cinder-854267-3-iscsi devstack@keystone.service[81425]: your server socket listen backlog is limited to 100 connections Aug 29 21:19:10.302853 dv-cinder-854267-3-iscsi devstack@keystone.service[81425]: your mercy for graceful operations on workers is 90 seconds Aug 29 21:19:10.303239 dv-cinder-854267-3-iscsi devstack@keystone.service[81425]: mapped 403077 bytes (393 KB) for 2 cores Aug 29 21:19:10.303317 dv-cinder-854267-3-iscsi devstack@keystone.service[81425]: *** Operational MODE: preforking *** Aug 29 21:19:10.303553 dv-cinder-854267-3-iscsi devstack@keystone.service[81425]: *** uWSGI is running in multiple interpreter mode *** Aug 29 21:19:10.303553 dv-cinder-854267-3-iscsi devstack@keystone.service[81425]: spawned uWSGI master process (pid: 81425) Aug 29 21:19:10.303757 dv-cinder-854267-3-iscsi systemd[1]: Started Devstack devstack@keystone.service. Aug 29 21:19:10.306915 dv-cinder-854267-3-iscsi devstack@keystone.service[81425]: spawned uWSGI worker 1 (pid: 81433, cores: 1) Aug 29 21:19:10.306915 dv-cinder-854267-3-iscsi devstack@keystone.service[81425]: spawned uWSGI worker 2 (pid: 81434, cores: 1) Aug 29 21:19:10.306915 dv-cinder-854267-3-iscsi devstack@keystone.service[81425]: running "unix_signal:15 gracefully_kill_them_all" (master-start)... Aug 29 21:19:14.596563 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Aug 29 21:19:14.596563 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: from cryptography.utils import int_from_bytes Aug 29 21:19:14.597372 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Aug 29 21:19:14.597372 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: from cryptography.utils import int_from_bytes Aug 29 21:19:14.599355 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Aug 29 21:19:14.599355 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: from cryptography.utils import int_from_bytes Aug 29 21:19:14.599824 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Aug 29 21:19:14.599824 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: from cryptography.utils import int_from_bytes Aug 29 21:19:14.647461 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Aug 29 21:19:14.647646 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Aug 29 21:19:14.647829 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] command line args: [] {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Aug 29 21:19:14.648101 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Aug 29 21:19:14.648280 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ================================================================================ {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Aug 29 21:19:14.648535 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] admin_token = **** {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 21:19:14.648782 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] config_dir = [] {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 21:19:14.648985 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 21:19:14.649153 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] config_source = [] {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 21:19:14.649334 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] debug = True {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 21:19:14.649534 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 21:19:14.650718 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 21:19:14.650718 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] insecure_debug = False {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 21:19:14.650718 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 21:19:14.650718 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 21:19:14.650976 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Aug 29 21:19:14.650976 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Aug 29 21:19:14.651102 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] list_limit = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 21:19:14.651163 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] command line args: [] {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Aug 29 21:19:14.651226 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] log_config_append = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 21:19:14.651226 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 21:19:14.651336 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Aug 29 21:19:14.651405 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] log_dir = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 21:19:14.651465 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ================================================================================ {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Aug 29 21:19:14.651524 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] log_file = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 21:19:14.651628 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] admin_token = **** {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 21:19:14.651756 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 21:19:14.651879 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] config_dir = [] {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 21:19:14.651995 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 21:19:14.652079 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 21:19:14.652155 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 21:19:14.652232 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] config_source = [] {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 21:19:14.652370 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 21:19:14.652475 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] debug = True {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 21:19:14.652562 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 21:19:14.652638 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 21:19:14.652743 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 21:19:14.652818 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 21:19:14.652941 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 21:19:14.653048 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] insecure_debug = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 21:19:14.653185 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 21:19:14.653290 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 21:19:14.653417 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 21:19:14.653511 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 21:19:14.653649 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 21:19:14.653744 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] list_limit = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 21:19:14.653876 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] log_config_append = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 21:19:14.653965 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 21:19:14.654037 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 21:19:14.654128 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 21:19:14.654230 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] log_dir = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 21:19:14.654363 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] log_file = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 21:19:14.654455 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 21:19:14.654602 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 21:19:14.654695 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] notification_format = cadf {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 21:19:14.654853 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 21:19:14.654952 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 21:19:14.655102 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 21:19:14.655340 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.90/identity {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 21:19:14.655435 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 21:19:14.655517 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] publish_errors = False {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 21:19:14.655590 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 21:19:14.655698 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 21:19:14.655797 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 21:19:14.655946 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 21:19:14.656042 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 21:19:14.656129 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 21:19:14.656201 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 21:19:14.656353 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 21:19:14.656445 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 21:19:14.656587 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 21:19:14.656733 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 21:19:14.656860 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] standard_threads = False {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 21:19:14.656969 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 21:19:14.657106 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] strict_password_check = False {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 21:19:14.657213 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 21:19:14.657320 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 21:19:14.657463 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] use_eventlog = False {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 21:19:14.657580 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 21:19:14.657688 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] use_journal = False {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 21:19:14.657847 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] notification_format = cadf {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 21:19:14.658038 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] use_json = False {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 21:19:14.658212 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 21:19:14.658357 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] use_stderr = False {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 21:19:14.658357 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] use_syslog = False {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 21:19:14.658596 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.90/identity {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 21:19:14.658596 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] publish_errors = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 21:19:14.658840 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] watch_log_file = False {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 21:19:14.658940 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 21:19:14.660135 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 21:19:14.660261 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.660377 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 21:19:14.660543 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.660661 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 21:19:14.660822 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.660944 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 21:19:14.660944 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] standard_threads = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 21:19:14.661167 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.661167 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.662336 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] strict_password_check = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 21:19:14.662462 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.662579 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 21:19:14.662693 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.662807 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] use_eventlog = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 21:19:14.662930 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.663059 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] use_journal = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 21:19:14.663059 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] use_json = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 21:19:14.663277 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.663391 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] use_stderr = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 21:19:14.663511 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.663637 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] use_syslog = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 21:19:14.663637 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] watch_log_file = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 21:19:14.663852 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.663966 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.664087 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.664202 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.664323 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.664445 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.664560 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.664674 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.664787 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.664899 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.665022 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.665149 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.665264 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.665378 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.665497 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.665618 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.665741 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.665879 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.666002 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.666117 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.666238 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.666352 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.666477 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.666592 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.666705 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.666819 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.666933 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.667052 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.667250 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.667371 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.667486 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.667600 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.667720 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.667843 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.667960 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.667960 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.668181 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.668297 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.668412 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.668533 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.668655 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.668769 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.668889 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.669039 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.669155 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.669277 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.669401 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.669520 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.669635 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.669747 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.669909 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.670037 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.670191 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.670308 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.670423 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.670537 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.670656 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.670779 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.670894 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.671008 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.671121 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.671240 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.671354 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.671477 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.671598 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.671713 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.671827 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.671946 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.672066 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.672192 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.672313 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.672428 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.672542 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.672660 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.672774 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.672967 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.673084 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.673199 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.673199 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.673412 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.673532 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.673657 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.673779 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.673912 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.674026 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.674149 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.674265 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.674387 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.674502 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.674616 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.674736 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.674836 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.674938 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.675050 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.675149 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.675244 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.675346 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.675449 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.675551 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.675656 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.675762 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.675859 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.675972 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.676053 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.676131 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.676211 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.676283 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.676350 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.676420 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.676492 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.676563 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.676638 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.676716 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.676789 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.676884 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.677941 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.678045 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.678117 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.678198 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.678271 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.678350 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.678423 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.678489 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.678560 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.678634 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.678707 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.678778 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.678850 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.678931 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] cache.enabled = True {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.678931 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.679069 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.679142 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.679142 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.679261 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.679334 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.679413 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.679494 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] cache.memcache_password = {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.679568 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.679692 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.679768 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.679880 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.679995 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.680125 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.680278 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.680396 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.680510 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.680625 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] cache.memcache_sasl_enabled = False {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.680740 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.680864 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.680982 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.681096 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.681210 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.681325 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] cache.memcache_username = {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.681446 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.681573 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.681690 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] cache.enabled = True {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.681821 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.681940 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.682056 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.682171 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.682298 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.682417 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.682533 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.682647 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.682762 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.682877 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] cache.memcache_password = {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.683001 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.683117 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.683232 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.683346 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.683468 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.683584 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.683774 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.683893 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.684009 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.684137 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] cache.memcache_sasl_enabled = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.684293 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.684424 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.684553 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.684686 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.684803 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.684919 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] cache.memcache_username = {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.685036 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.685150 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.685271 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.685398 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.685515 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.685630 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.685748 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.685916 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.686041 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] auth.external = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.686168 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.686285 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] auth.mapped = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.686397 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.686512 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.686629 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.686743 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.686866 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.686983 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] auth.password = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.687098 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.687215 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] auth.token = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.687338 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.687464 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.687615 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.687741 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] catalog.caching = True {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.687867 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.687992 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.688110 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.688227 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.688343 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.688458 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.688586 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.688703 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.688820 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.688951 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.689066 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.689182 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] credential.caching = True {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.689308 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.689426 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] credential.driver = sql {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.689542 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] auth.external = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.689657 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.689773 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] auth.mapped = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.689905 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.689905 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.690138 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.690138 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.690394 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.690512 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] auth.password = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.690636 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.690762 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] auth.token = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.690963 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.691116 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.691236 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.691354 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] catalog.caching = True {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.691470 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.691596 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.691713 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.691830 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.691946 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.692063 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.692179 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.692304 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.692423 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.692619 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.693074 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.693202 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] credential.caching = True {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.693321 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.693448 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] credential.driver = sql {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.693566 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] federation.caching = True {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.693682 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.693855 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.693975 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.694092 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] federation.driver = sql {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.694221 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.694340 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.694457 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.694574 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.694690 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.694806 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.694932 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.695049 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.695165 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.695282 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.695398 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.695515 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.695639 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.695756 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.695873 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.695993 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.696109 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.696233 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.696351 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.696467 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] identity.caching = True {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.696583 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.696699 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.696699 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.696925 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.697043 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.697160 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] federation.caching = True {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.697275 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.697390 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.697506 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] identity.driver = sql {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.697632 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] federation.driver = sql {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.697758 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.697900 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.698018 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.698134 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.698250 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.698442 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.698563 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.698680 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.698796 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.698912 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.699029 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.699153 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.699282 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.699391 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.699506 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.699628 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.699745 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.699869 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.699986 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.700103 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] identity.caching = True {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.700219 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.700335 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.700483 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.700611 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.700729 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.700846 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.700961 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.701077 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.701192 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.701316 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] identity.driver = sql {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.701434 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.701550 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.701666 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.701800 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.701919 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.702048 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.702166 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.702283 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.702398 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.702514 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.702629 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.702757 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.702876 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.702992 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.703108 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.703224 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.703339 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.703463 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.703580 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.703580 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.703797 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.703913 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.704029 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.704154 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.704272 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.704388 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.704503 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.704618 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.704734 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.704858 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.password = **** {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.704976 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.705098 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.705216 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.705331 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.705448 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.705572 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.705690 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.705838 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.705957 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.706075 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.706190 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.706318 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.706443 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.706560 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.706676 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.706791 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.706791 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.707018 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.707136 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.707251 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.707367 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.707483 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.707606 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.707723 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.707838 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.707954 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.708113 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.708303 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.708481 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.user = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.708605 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.708722 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.708981 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.709099 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.709218 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.password = **** {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.709343 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.709462 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.709578 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.709694 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.709830 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.709949 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.710078 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.710199 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.710316 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.710432 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.710579 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.710697 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.710822 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.710941 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.711057 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.711173 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.711290 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.711406 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.711530 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.711648 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.711764 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.711880 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.711997 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.712113 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.712236 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.712356 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.712471 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.712587 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.712702 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.712818 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.user = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.712941 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.713060 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.713176 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.713293 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.713410 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.713527 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.713651 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.713769 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.713905 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.714022 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.714138 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.714254 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.714382 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.714502 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.714619 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.714735 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.714851 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.714967 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.715091 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] policy.driver = sql {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.715209 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.715324 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.715449 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.715565 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.715681 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.715810 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.715928 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.716045 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] receipt.caching = True {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.716161 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.716283 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.716400 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.716400 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.716630 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.716630 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.716848 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.716971 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.717088 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.717212 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.717329 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.717451 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] resource.caching = True {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.717570 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.717685 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.717854 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.717854 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.718090 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] resource.driver = sql {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.718207 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.718323 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.718439 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.718555 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.718681 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.718798 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.718915 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.719030 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] revoke.caching = True {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.719146 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] policy.driver = sql {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.719262 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.719387 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.719504 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.719621 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.719737 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] role.cache_time = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.719853 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.719969 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] role.caching = True {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.720097 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] receipt.caching = True {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.720215 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] role.driver = sql {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.720331 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.720737 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] role.list_limit = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.720859 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.720976 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.721100 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.721218 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.721335 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.721451 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.721566 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.721756 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.721895 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] resource.caching = True {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.722013 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.722130 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.722245 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.722361 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] resource.driver = sql {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.722489 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.722609 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.722725 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.722842 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.722957 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.723080 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.723206 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.723206 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.723426 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] revoke.caching = True {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.723541 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.724178 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.724374 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.724755 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.724755 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] role.cache_time = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.725026 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.725153 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] role.caching = True {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.725276 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.725397 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] role.driver = sql {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.725527 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.725648 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] role.list_limit = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.725767 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.725908 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.726028 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.726145 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.726277 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.726400 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.726518 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.726642 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.726762 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.726880 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.727008 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.727128 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.727245 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.727363 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.727480 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.727597 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.727722 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.727842 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.727960 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.728079 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.728255 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.728379 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.728494 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.728612 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.728730 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.728847 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.728965 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.728965 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.729205 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.729324 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] token.cache_time = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.729442 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.729559 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] token.caching = True {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.729676 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.729895 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.730018 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.730139 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] token.provider = fernet {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.730256 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.730374 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.730491 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.730622 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.730780 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.730903 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.731021 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.731138 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.731257 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.731387 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.731508 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.731626 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.731744 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.731862 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] trust.driver = sql {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.731979 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.732110 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.732110 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.732335 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.732453 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.732570 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.732687 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.732813 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.732933 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.733051 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.733168 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.733296 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.733424 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.733544 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.733669 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.733808 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] token.cache_time = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.733928 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] database.connection = **** {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.734047 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] token.caching = True {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.734165 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.734282 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.734414 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.734535 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] token.provider = fernet {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.734657 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.734776 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.734901 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.735020 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.735146 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.735266 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.735384 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.735502 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.735620 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.735738 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.735864 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.735984 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.736102 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.736220 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] trust.driver = sql {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.736338 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.736457 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.736583 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.736702 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.736820 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.736937 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.737055 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.737173 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.737298 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.737418 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.737537 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.737654 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.737780 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.737930 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.738034 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.738151 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.738278 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.738278 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Aug 29 21:19:14.738501 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] database.connection = **** {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.738501 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.738501 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.738501 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.738501 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.738501 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.738501 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.738501 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.738501 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.740033 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.740033 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.740033 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.740033 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.740033 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.740033 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.740033 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.740033 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.740033 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.741053 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 21:19:14.741053 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Aug 29 21:19:14.764099 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=81434) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 21:19:14.764325 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=81434) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 21:19:14.764547 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=81434) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 21:19:14.767691 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=81433) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 21:19:14.767930 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=81433) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 21:19:14.768159 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=81433) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 21:19:14.772996 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=81434) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 21:19:14.777073 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=81433) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 21:19:14.801749 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=81434) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 21:19:14.802949 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=81434) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 21:19:14.804875 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=81433) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 21:19:14.805111 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=81433) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 21:19:14.859230 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=81434) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 21:19:14.860428 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=81433) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 21:19:14.908130 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=81434) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 21:19:14.908367 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=81434) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 21:19:14.909442 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=81433) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 21:19:14.909686 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=81433) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 21:19:15.065131 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81434) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 21:19:15.065359 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=81434) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 21:19:15.065582 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=81434) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 21:19:15.065673 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81433) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 21:19:15.065831 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=81434) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 21:19:15.065962 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=81433) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 21:19:15.066049 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=81434) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 21:19:15.066146 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=81433) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 21:19:15.066255 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=81434) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 21:19:15.066343 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=81433) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 21:19:15.066506 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=81433) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 21:19:15.066728 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=81433) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 21:19:15.089894 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81434) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 21:19:15.090129 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=81434) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 21:19:15.090354 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=81434) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 21:19:15.090468 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81433) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 21:19:15.090577 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=81434) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 21:19:15.090711 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=81433) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 21:19:15.090806 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=81434) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 21:19:15.090949 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=81433) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 21:19:15.091058 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=81434) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 21:19:15.091175 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=81433) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 21:19:15.091265 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81434) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 21:19:15.091392 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=81433) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 21:19:15.091609 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=81433) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 21:19:15.091829 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81433) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 21:19:15.103224 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=81434) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 21:19:15.104466 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=81433) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 21:19:15.116647 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81434) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 21:19:15.116822 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=81434) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 21:19:15.117000 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81433) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 21:19:15.117087 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=81434) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 21:19:15.117244 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=81433) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 21:19:15.117340 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=81434) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 21:19:15.117454 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=81434) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 21:19:15.117538 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=81433) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 21:19:15.117667 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=81434) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 21:19:15.117749 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=81433) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 21:19:15.117904 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81434) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 21:19:15.118003 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=81433) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 21:19:15.118148 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=81433) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 21:19:15.118367 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81433) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 21:19:15.135792 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.135991 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Aug 29 21:19:15.136162 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.136334 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.136419 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Aug 29 21:19:15.136535 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Aug 29 21:19:15.136718 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.136802 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.136927 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Aug 29 21:19:15.137054 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.138043 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.138043 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.138204 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.138204 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.138363 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.138514 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.147721 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.147995 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.148245 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.148345 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.148534 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.148618 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.148808 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.149097 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.155575 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=81434) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 29 21:19:15.158050 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=81433) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 29 21:19:15.158471 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=81434) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 29 21:19:15.158753 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.159000 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=81433) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 29 21:19:15.159101 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.159294 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.159376 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.159576 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.159660 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.159838 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.159930 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.160106 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.160188 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.160380 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.160461 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.161141 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.161299 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.161397 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.161473 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.161542 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.161626 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.161874 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.162180 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.184584 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.185777 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.186059 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=81434) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 29 21:19:15.186345 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.187271 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=81433) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 29 21:19:15.187562 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.190924 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=81434) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 29 21:19:15.191210 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.191490 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.192239 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=81433) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 29 21:19:15.193104 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.193408 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.198691 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=81434) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 29 21:19:15.198980 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.199800 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=81433) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 29 21:19:15.200092 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.202913 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=81434) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 29 21:19:15.203210 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.203494 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.203779 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.204056 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.204186 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=81433) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 29 21:19:15.204487 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.204778 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.205066 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.205349 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.205665 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.205969 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.206074 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.206248 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.206366 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.206653 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.222375 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.222655 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.224483 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.224768 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.226652 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=81434) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 29 21:19:15.228673 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=81433) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 29 21:19:15.229689 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.230006 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.231999 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.232303 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.234249 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=81434) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 29 21:19:15.236555 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=81433) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 29 21:19:15.237357 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=81434) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 29 21:19:15.240259 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=81433) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 29 21:19:15.241345 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.241345 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.241867 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.241974 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.242258 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.242953 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.243787 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.244096 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.244391 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.244693 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.244994 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.245280 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.259056 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=81434) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 29 21:19:15.259345 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.259614 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.259874 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.261902 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=81433) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 29 21:19:15.262199 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.262472 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.262752 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.266579 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH2: ['/token' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.268222 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.269618 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH2: ['/token' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.269752 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.270057 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.271745 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.272983 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=81434) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 29 21:19:15.273270 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.273353 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.273560 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.273641 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.273872 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.274165 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.276254 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=81433) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 29 21:19:15.276549 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.276954 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.277508 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.277844 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.285675 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=81434) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 29 21:19:15.285980 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.286842 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.287129 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.287411 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.287691 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.287990 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.289694 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=81433) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 29 21:19:15.290035 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.290430 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.290723 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.291014 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.291302 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.291603 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.303887 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=81434) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 29 21:19:15.306916 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=81434) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 29 21:19:15.307791 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=81433) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 29 21:19:15.309777 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.310748 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=81433) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 29 21:19:15.311219 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.312685 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=81434) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 29 21:19:15.312967 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.313260 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.313636 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.315101 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.316719 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=81433) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 29 21:19:15.317019 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.317314 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.319795 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.321270 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=81434) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 29 21:19:15.323989 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.324135 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.324436 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.324725 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.325011 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.325741 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=81433) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 29 21:19:15.328767 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.329084 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.329402 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.329666 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.332992 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=81434) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 29 21:19:15.333429 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.333722 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.337581 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=81433) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 29 21:19:15.337906 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.338200 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.340590 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=81434) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 29 21:19:15.340875 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.341156 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.341434 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.341708 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.342012 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.342286 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.342570 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.342847 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.343121 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.343390 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.343653 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.343926 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.344210 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.345107 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=81433) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 29 21:19:15.345399 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.345978 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.346263 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.346542 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.346820 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.347097 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.347378 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.347661 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.347940 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.348214 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.348477 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.349281 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.349796 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 21:19:15.379402 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: WSGI app 0 (mountpoint='') ready in 5 seconds on interpreter 0x55d33ffeb9c0 pid: 81434 (default app) Aug 29 21:19:15.382922 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-257e8273-8ff5-4cdf-97a0-33f091912fd9 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:15.383206 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-257e8273-8ff5-4cdf-97a0-33f091912fd9 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:15.383479 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-257e8273-8ff5-4cdf-97a0-33f091912fd9 None None] PATH_INFO: `/v3` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:15.384619 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: WSGI app 0 (mountpoint='') ready in 5 seconds on interpreter 0x55d33ffeb9c0 pid: 81433 (default app) Aug 29 21:19:15.387653 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1/1] 10.222.0.90 () {50 vars in 791 bytes} [Mon Aug 29 21:19:15 2022] GET /identity/v3/ => generated 255 bytes in 8 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:19:21.986520 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c16a373-72d0-4cca-83a9-50cf5a2ddfbf None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:21.987059 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c16a373-72d0-4cca-83a9-50cf5a2ddfbf None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:21.987059 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c16a373-72d0-4cca-83a9-50cf5a2ddfbf None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:21.992047 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1/2] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:19:21 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 29 21:19:21.997423 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb1a14b-8787-4ff6-bacc-e6f4ab8e6630 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:21.998004 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb1a14b-8787-4ff6-bacc-e6f4ab8e6630 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:21.998004 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb1a14b-8787-4ff6-bacc-e6f4ab8e6630 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:22.069932 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: INFO dbcounter [None req-7eb1a14b-8787-4ff6-bacc-e6f4ab8e6630 None None] Registered counter for database keystone Aug 29 21:19:22.088815 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG oslo_db.sqlalchemy.engines [None req-7eb1a14b-8787-4ff6-bacc-e6f4ab8e6630 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=81434) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Aug 29 21:19:22.095058 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writer thread running {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Aug 29 21:19:22.279990 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG passlib.handlers.bcrypt [None req-7eb1a14b-8787-4ff6-bacc-e6f4ab8e6630 None None] detected 'bcrypt' backend, version '4.0.0' {{(pid=81434) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Aug 29 21:19:22.280292 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG passlib.handlers.bcrypt [None req-7eb1a14b-8787-4ff6-bacc-e6f4ab8e6630 None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=81434) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Aug 29 21:19:22.340760 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-7eb1a14b-8787-4ff6-bacc-e6f4ab8e6630 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:19:22.342407 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-7eb1a14b-8787-4ff6-bacc-e6f4ab8e6630 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:19:22.484419 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 2/3] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:19:21 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 488 msecs (HTTP/1.1 201) 6 headers in 384 bytes (2 switches on core 0) Aug 29 21:19:22.990248 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c16a373-72d0-4cca-83a9-50cf5a2ddfbf None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:23.060613 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: INFO dbcounter [None req-3c16a373-72d0-4cca-83a9-50cf5a2ddfbf None None] Registered counter for database keystone Aug 29 21:19:23.082416 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG oslo_db.sqlalchemy.engines [None req-3c16a373-72d0-4cca-83a9-50cf5a2ddfbf None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=81433) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Aug 29 21:19:23.089386 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writer thread running {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Aug 29 21:19:23.210470 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c16a373-72d0-4cca-83a9-50cf5a2ddfbf None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:23.214977 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-240b2322-5548-405d-822f-3e8759b5c772 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:23.215603 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-240b2322-5548-405d-822f-3e8759b5c772 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:23.215884 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-240b2322-5548-405d-822f-3e8759b5c772 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:23.216157 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-240b2322-5548-405d-822f-3e8759b5c772 admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:23.224533 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-240b2322-5548-405d-822f-3e8759b5c772 admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:23.277556 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: WARNING keystone.common.rbac_enforcer.enforcer [None req-240b2322-5548-405d-822f-3e8759b5c772 admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Aug 29 21:19:23.277925 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-240b2322-5548-405d-822f-3e8759b5c772 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:23.288171 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 2/4] 10.222.0.90 () {58 vars in 1324 bytes} [Mon Aug 29 21:19:22 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 299 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 29 21:19:23.294165 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eb1a14b-8787-4ff6-bacc-e6f4ab8e6630 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:23.309068 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eb1a14b-8787-4ff6-bacc-e6f4ab8e6630 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:23.313663 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2931bccb-58ce-4fe6-93dc-27d906a5c104 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:23.314801 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2931bccb-58ce-4fe6-93dc-27d906a5c104 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:23.315083 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2931bccb-58ce-4fe6-93dc-27d906a5c104 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:23.315355 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2931bccb-58ce-4fe6-93dc-27d906a5c104 admin admin] PATH_INFO: `/v3/projects/fa1ebaa0d49241599f614eb2efe4d8c8` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:23.316953 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2931bccb-58ce-4fe6-93dc-27d906a5c104 admin admin] RBAC: Authorizing `identity:get_project(project_id=fa1ebaa0d49241599f614eb2efe4d8c8)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:23.371498 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: WARNING keystone.common.rbac_enforcer.enforcer [None req-2931bccb-58ce-4fe6-93dc-27d906a5c104 admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Aug 29 21:19:23.371831 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2931bccb-58ce-4fe6-93dc-27d906a5c104 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:23.376514 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 3/5] 10.222.0.90 () {56 vars in 1207 bytes} [Mon Aug 29 21:19:23 2022] GET /identity/v3/projects/fa1ebaa0d49241599f614eb2efe4d8c8 => generated 344 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:19:25.243263 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2682e9d4-8a76-419a-a3ce-861279e3f1cc None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:25.243768 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2682e9d4-8a76-419a-a3ce-861279e3f1cc None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:25.243768 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2682e9d4-8a76-419a-a3ce-861279e3f1cc None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:25.247816 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 3/6] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:19:25 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 29 21:19:25.253197 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a365034d-1212-4648-afdd-d47426bcc5b8 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:25.253581 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a365034d-1212-4648-afdd-d47426bcc5b8 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:25.253755 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a365034d-1212-4648-afdd-d47426bcc5b8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:25.274180 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-a365034d-1212-4648-afdd-d47426bcc5b8 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:19:25.275731 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-a365034d-1212-4648-afdd-d47426bcc5b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:19:25.283653 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 4/7] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:19:25 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Aug 29 21:19:25.557646 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2682e9d4-8a76-419a-a3ce-861279e3f1cc None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:25.570184 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2682e9d4-8a76-419a-a3ce-861279e3f1cc None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:25.574429 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-104a1fc6-9228-4091-b592-27150b72e9b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:25.575035 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-104a1fc6-9228-4091-b592-27150b72e9b5 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:25.575319 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-104a1fc6-9228-4091-b592-27150b72e9b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:25.575591 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-104a1fc6-9228-4091-b592-27150b72e9b5 admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:25.584319 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-104a1fc6-9228-4091-b592-27150b72e9b5 admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:25.586125 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-104a1fc6-9228-4091-b592-27150b72e9b5 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:25.594844 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 4/8] 10.222.0.90 () {58 vars in 1324 bytes} [Mon Aug 29 21:19:25 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 29 21:19:25.601157 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a365034d-1212-4648-afdd-d47426bcc5b8 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:25.610300 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a365034d-1212-4648-afdd-d47426bcc5b8 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:25.615076 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbf5e4dd-67dd-4da4-8dab-aa34d98cd547 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:25.615708 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbf5e4dd-67dd-4da4-8dab-aa34d98cd547 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:25.615994 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbf5e4dd-67dd-4da4-8dab-aa34d98cd547 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:25.616271 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbf5e4dd-67dd-4da4-8dab-aa34d98cd547 admin admin] PATH_INFO: `/v3/users/97a0b159abf44cd59fe5a2b7bde28adf` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:25.617586 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbf5e4dd-67dd-4da4-8dab-aa34d98cd547 admin admin] RBAC: Authorizing `identity:get_user(user_id=97a0b159abf44cd59fe5a2b7bde28adf)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:25.619451 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbf5e4dd-67dd-4da4-8dab-aa34d98cd547 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:25.620973 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 5/9] 10.222.0.90 () {56 vars in 1198 bytes} [Mon Aug 29 21:19:25 2022] GET /identity/v3/users/97a0b159abf44cd59fe5a2b7bde28adf => generated 247 bytes in 20 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:19:27.577108 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53e6bbf8-788d-402b-b3c4-b78ff677ae8d None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:27.577651 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53e6bbf8-788d-402b-b3c4-b78ff677ae8d None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:27.577651 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53e6bbf8-788d-402b-b3c4-b78ff677ae8d None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:27.579234 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 5/10] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:19:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:19:27.584594 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-724794d7-7987-46a7-bdd9-2947b4dd5a00 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:27.585006 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-724794d7-7987-46a7-bdd9-2947b4dd5a00 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:27.585225 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-724794d7-7987-46a7-bdd9-2947b4dd5a00 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:27.604747 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-724794d7-7987-46a7-bdd9-2947b4dd5a00 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:19:27.605965 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-724794d7-7987-46a7-bdd9-2947b4dd5a00 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:19:27.634210 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5330295-fe89-444d-946e-4a5924f5c7e4 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:27.634526 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5330295-fe89-444d-946e-4a5924f5c7e4 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:27.634842 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5330295-fe89-444d-946e-4a5924f5c7e4 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:27.636312 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 6/11] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:19:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 29 21:19:27.641316 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40ab0c1a-f59b-46a7-9547-0ac35e6f9c67 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:27.641627 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40ab0c1a-f59b-46a7-9547-0ac35e6f9c67 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:27.642020 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40ab0c1a-f59b-46a7-9547-0ac35e6f9c67 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:27.684974 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG passlib.handlers.bcrypt [None req-40ab0c1a-f59b-46a7-9547-0ac35e6f9c67 None None] detected 'bcrypt' backend, version '4.0.0' {{(pid=81433) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Aug 29 21:19:27.685369 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG passlib.handlers.bcrypt [None req-40ab0c1a-f59b-46a7-9547-0ac35e6f9c67 None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=81433) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Aug 29 21:19:27.688841 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 6/12] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:19:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 21:19:27.735716 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-40ab0c1a-f59b-46a7-9547-0ac35e6f9c67 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:19:27.737382 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-40ab0c1a-f59b-46a7-9547-0ac35e6f9c67 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:19:27.838753 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 7/13] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:19:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 198 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 29 21:19:27.952453 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-724794d7-7987-46a7-bdd9-2947b4dd5a00 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:27.979910 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:28.060812 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-724794d7-7987-46a7-bdd9-2947b4dd5a00 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:28.095307 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb1d724e-2e1b-4bff-a63d-60f147343afe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:28.096134 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb1d724e-2e1b-4bff-a63d-60f147343afe None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:28.096419 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb1d724e-2e1b-4bff-a63d-60f147343afe None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:28.096696 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb1d724e-2e1b-4bff-a63d-60f147343afe None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:28.106511 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40ab0c1a-f59b-46a7-9547-0ac35e6f9c67 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:28.108588 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb1d724e-2e1b-4bff-a63d-60f147343afe None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 21:19:28.117432 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-eb1d724e-2e1b-4bff-a63d-60f147343afe None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 21:19:28.117432 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:19:28.117432 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:19:28.117432 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:19:28.117432 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:19:28.117432 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:19:28.117432 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:19:28.117432 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:19:28.117432 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:19:28.117432 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:19:28.117432 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:19:28.117432 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:19:28.117432 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 21:19:28.117432 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 21:19:28.117432 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 21:19:28.117432 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 21:19:28.118613 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 21:19:28.118613 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 21:19:28.118613 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 21:19:28.118613 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 21:19:28.118613 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:19:28.118613 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:19:28.118613 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:19:28.118613 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:19:28.118613 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:19:28.118613 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:19:28.118613 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:19:28.118613 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:19:28.118613 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:19:28.118613 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:19:28.118613 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:19:28.118613 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:19:28.119454 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:19:28.119454 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:19:28.119454 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:19:28.119454 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:19:28.119454 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 21:19:28.119454 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 21:19:28.119454 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 21:19:28.119454 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 21:19:28.119454 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 21:19:28.119454 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 21:19:28.119454 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 21:19:28.119454 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:19:28.119454 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 7/14] 10.222.0.90 () {56 vars in 1096 bytes} [Mon Aug 29 21:19:27 2022] GET /identity/v3/roles/admin => generated 83 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:19:28.124253 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb1d724e-2e1b-4bff-a63d-60f147343afe None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:28.204955 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:28.208971 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40ab0c1a-f59b-46a7-9547-0ac35e6f9c67 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:28.211476 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:28.225161 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb1d724e-2e1b-4bff-a63d-60f147343afe None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:28.247654 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7641f597-4329-4eb1-b44b-db0d0beec13a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:28.248541 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7641f597-4329-4eb1-b44b-db0d0beec13a None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:28.248831 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7641f597-4329-4eb1-b44b-db0d0beec13a None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:28.249109 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7641f597-4329-4eb1-b44b-db0d0beec13a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:28.259032 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90248d5a-efc0-4be3-8146-2f2cb49ecefa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:28.259646 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90248d5a-efc0-4be3-8146-2f2cb49ecefa None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:28.259928 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90248d5a-efc0-4be3-8146-2f2cb49ecefa None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:28.260203 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90248d5a-efc0-4be3-8146-2f2cb49ecefa None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:28.260873 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90248d5a-efc0-4be3-8146-2f2cb49ecefa None admin] RBAC: Adding query filter params (name=admin) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:19:28.261175 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90248d5a-efc0-4be3-8146-2f2cb49ecefa None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:28.262608 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90248d5a-efc0-4be3-8146-2f2cb49ecefa None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:28.274856 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 8/15] 10.222.0.90 () {56 vars in 1099 bytes} [Mon Aug 29 21:19:28 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:19:28.280373 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90248d5a-efc0-4be3-8146-2f2cb49ecefa None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:28.318163 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7641f597-4329-4eb1-b44b-db0d0beec13a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:28.319717 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7641f597-4329-4eb1-b44b-db0d0beec13a None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:28.379659 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90248d5a-efc0-4be3-8146-2f2cb49ecefa None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:28.414141 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2407e82f-0d79-409b-b8ac-65d692c070a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:28.414917 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2407e82f-0d79-409b-b8ac-65d692c070a4 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:28.415251 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2407e82f-0d79-409b-b8ac-65d692c070a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:28.415699 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2407e82f-0d79-409b-b8ac-65d692c070a4 None admin] PATH_INFO: `/v3/users/97a0b159abf44cd59fe5a2b7bde28adf` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:28.417218 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2407e82f-0d79-409b-b8ac-65d692c070a4 None admin] RBAC: Authorizing `identity:get_user(user_id=97a0b159abf44cd59fe5a2b7bde28adf)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:28.418933 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 8/16] 10.222.0.90 () {58 vars in 1282 bytes} [Mon Aug 29 21:19:28 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 313 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Aug 29 21:19:28.419040 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2407e82f-0d79-409b-b8ac-65d692c070a4 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:28.420679 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 9/17] 10.222.0.90 () {56 vars in 1177 bytes} [Mon Aug 29 21:19:28 2022] GET /identity/v3/users/97a0b159abf44cd59fe5a2b7bde28adf => generated 247 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:19:28.425918 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2407e82f-0d79-409b-b8ac-65d692c070a4 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:28.426168 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7641f597-4329-4eb1-b44b-db0d0beec13a None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:28.429382 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:28.441150 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:28.527135 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7641f597-4329-4eb1-b44b-db0d0beec13a None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:28.527643 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2407e82f-0d79-409b-b8ac-65d692c070a4 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:28.561406 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ca7a649-18ed-4ed1-9247-246151eaa97c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:28.562554 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ca7a649-18ed-4ed1-9247-246151eaa97c None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:28.562838 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ca7a649-18ed-4ed1-9247-246151eaa97c None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:28.563141 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ca7a649-18ed-4ed1-9247-246151eaa97c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:28.563920 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8442f10b-38fa-46dd-954a-d4d4c66a43a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:28.564557 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8442f10b-38fa-46dd-954a-d4d4c66a43a3 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:28.564838 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8442f10b-38fa-46dd-954a-d4d4c66a43a3 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:28.565123 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8442f10b-38fa-46dd-954a-d4d4c66a43a3 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:28.566276 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8442f10b-38fa-46dd-954a-d4d4c66a43a3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:28.568266 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8442f10b-38fa-46dd-954a-d4d4c66a43a3 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:28.569949 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 10/18] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:19:28 2022] GET /identity/v3/domains/default => generated 199 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:19:28.575655 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8442f10b-38fa-46dd-954a-d4d4c66a43a3 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:28.582598 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ca7a649-18ed-4ed1-9247-246151eaa97c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:28.584735 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ca7a649-18ed-4ed1-9247-246151eaa97c None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:28.604225 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-8ca7a649-18ed-4ed1-9247-246151eaa97c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:19:28.604225 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:19:28.604225 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:19:28.604225 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:19:28.604225 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:19:28.604225 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:19:28.604225 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:19:28.604225 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:19:28.604225 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:19:28.604225 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:19:28.604225 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:19:28.604225 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:19:28.604225 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 21:19:28.604225 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 21:19:28.604225 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 21:19:28.604225 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 21:19:28.605123 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:19:28.605123 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:19:28.605123 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:19:28.605123 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:19:28.605123 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:19:28.605123 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:19:28.605123 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:19:28.605123 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:19:28.605123 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:19:28.605123 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:19:28.605123 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:19:28.605123 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:19:28.605123 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:19:28.605123 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:19:28.605123 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:19:28.605123 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:19:28.605123 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 21:19:28.605979 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 21:19:28.605979 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:19:28.605979 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:19:28.605979 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 9/19] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:19:28 2022] GET /identity/v3/domains/Default => generated 87 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:19:28.611052 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ca7a649-18ed-4ed1-9247-246151eaa97c None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:28.657816 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:28.666969 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:28.678622 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8442f10b-38fa-46dd-954a-d4d4c66a43a3 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:28.712505 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b91026a9-2432-407d-8f2a-ac936d263fa4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:28.712935 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ca7a649-18ed-4ed1-9247-246151eaa97c None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:28.713116 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b91026a9-2432-407d-8f2a-ac936d263fa4 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:28.713397 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b91026a9-2432-407d-8f2a-ac936d263fa4 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:28.713666 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b91026a9-2432-407d-8f2a-ac936d263fa4 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:28.714428 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b91026a9-2432-407d-8f2a-ac936d263fa4 None admin] RBAC: Adding query filter params (role.id=a6e562cdc96c4be691d638990f8403fd, scope.domain.id=default, user.id=97a0b159abf44cd59fe5a2b7bde28adf) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:19:28.714735 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b91026a9-2432-407d-8f2a-ac936d263fa4 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:28.716173 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b91026a9-2432-407d-8f2a-ac936d263fa4 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:28.723230 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 11/20] 10.222.0.90 () {56 vars in 1322 bytes} [Mon Aug 29 21:19:28 2022] GET /identity/v3/role_assignments?user.id=97a0b159abf44cd59fe5a2b7bde28adf&scope.domain.id=default&role.id=a6e562cdc96c4be691d638990f8403fd => generated 233 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:19:28.749043 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afb00336-193c-4cd9-9133-a219d23da2fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:28.749680 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afb00336-193c-4cd9-9133-a219d23da2fd None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:28.750001 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afb00336-193c-4cd9-9133-a219d23da2fd None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:28.750281 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afb00336-193c-4cd9-9133-a219d23da2fd None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:28.750953 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afb00336-193c-4cd9-9133-a219d23da2fd None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:19:28.751259 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afb00336-193c-4cd9-9133-a219d23da2fd None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:28.752687 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afb00336-193c-4cd9-9133-a219d23da2fd None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:28.772864 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 10/21] 10.222.0.90 () {56 vars in 1109 bytes} [Mon Aug 29 21:19:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:19:30.791330 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90275115-a840-4609-8c9e-b6357990373d None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:30.791850 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90275115-a840-4609-8c9e-b6357990373d None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:30.791949 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90275115-a840-4609-8c9e-b6357990373d None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:30.793222 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 12/22] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:19:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:19:30.798744 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9699f68-859a-4f39-9d6b-b53a7af4e912 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:30.799178 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9699f68-859a-4f39-9d6b-b53a7af4e912 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:30.799368 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9699f68-859a-4f39-9d6b-b53a7af4e912 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:30.826902 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-e9699f68-859a-4f39-9d6b-b53a7af4e912 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:19:30.828144 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-e9699f68-859a-4f39-9d6b-b53a7af4e912 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:19:30.930671 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 11/23] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:19:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 21:19:30.996923 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dbe5b20-cf94-4783-93b3-4c9c16d7bc63 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:30.997235 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dbe5b20-cf94-4783-93b3-4c9c16d7bc63 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:30.997486 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dbe5b20-cf94-4783-93b3-4c9c16d7bc63 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:30.999153 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 13/24] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:19:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:19:31.005286 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faca5829-d737-4024-a044-0d9e8726df65 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:31.005581 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faca5829-d737-4024-a044-0d9e8726df65 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:31.005897 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faca5829-d737-4024-a044-0d9e8726df65 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:31.034154 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-faca5829-d737-4024-a044-0d9e8726df65 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:19:31.035358 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-faca5829-d737-4024-a044-0d9e8726df65 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:19:31.090864 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:31.124478 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 12/25] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:19:31 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 21:19:31.264212 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-120e0d0c-6391-48cc-988e-b96ce99df2cb None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:31.264524 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-120e0d0c-6391-48cc-988e-b96ce99df2cb None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:31.264771 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-120e0d0c-6391-48cc-988e-b96ce99df2cb None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:31.266321 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 14/26] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:19:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:19:31.272095 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7782ef8-1cec-466b-a27d-8cb61c36e37b None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:31.272384 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7782ef8-1cec-466b-a27d-8cb61c36e37b None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:31.272659 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7782ef8-1cec-466b-a27d-8cb61c36e37b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:31.293750 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-d7782ef8-1cec-466b-a27d-8cb61c36e37b None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:19:31.294966 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-d7782ef8-1cec-466b-a27d-8cb61c36e37b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:19:31.390369 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 13/27] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:19:31 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 21:19:31.421171 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-120e0d0c-6391-48cc-988e-b96ce99df2cb None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:31.546457 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-120e0d0c-6391-48cc-988e-b96ce99df2cb None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:31.585480 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-239b6c72-3282-404f-9fdf-a3f591ec56bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:31.586165 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-239b6c72-3282-404f-9fdf-a3f591ec56bc None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:31.586450 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-239b6c72-3282-404f-9fdf-a3f591ec56bc None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:31.586723 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-239b6c72-3282-404f-9fdf-a3f591ec56bc None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:31.590906 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7782ef8-1cec-466b-a27d-8cb61c36e37b None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:31.601618 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-239b6c72-3282-404f-9fdf-a3f591ec56bc None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 21:19:31.607638 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-239b6c72-3282-404f-9fdf-a3f591ec56bc None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 21:19:31.607638 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:19:31.607638 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:19:31.607638 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:19:31.607638 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:19:31.607638 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:19:31.607638 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:19:31.607638 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:19:31.607638 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:19:31.607638 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:19:31.607638 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:19:31.607638 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:19:31.607638 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 21:19:31.607638 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 21:19:31.607638 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 21:19:31.607638 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 21:19:31.608524 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 21:19:31.608524 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 21:19:31.608524 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 21:19:31.608524 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 21:19:31.608524 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:19:31.608524 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:19:31.608524 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:19:31.608524 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:19:31.608524 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:19:31.608524 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:19:31.608524 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:19:31.608524 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:19:31.608524 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:19:31.608524 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:19:31.608524 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:19:31.608524 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:19:31.609323 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:19:31.609323 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:19:31.609323 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:19:31.609323 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:19:31.609323 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 21:19:31.609323 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 21:19:31.609323 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 21:19:31.609323 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 21:19:31.609323 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 21:19:31.609323 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 21:19:31.609323 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 21:19:31.609323 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:19:31.609935 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 15/28] 10.222.0.90 () {56 vars in 1096 bytes} [Mon Aug 29 21:19:31 2022] GET /identity/v3/roles/admin => generated 83 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:19:31.615775 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-239b6c72-3282-404f-9fdf-a3f591ec56bc None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:31.642242 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:31.788270 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7782ef8-1cec-466b-a27d-8cb61c36e37b None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:31.792624 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-239b6c72-3282-404f-9fdf-a3f591ec56bc None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:31.802746 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:31.864061 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d42fe5c4-6caf-4933-8c83-f392057a1e70 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:31.865375 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d42fe5c4-6caf-4933-8c83-f392057a1e70 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:31.868965 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d42fe5c4-6caf-4933-8c83-f392057a1e70 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:31.869257 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d42fe5c4-6caf-4933-8c83-f392057a1e70 None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:31.869551 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca1b1ce2-367c-4356-909d-fb63ca7e5a77 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:31.869972 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d42fe5c4-6caf-4933-8c83-f392057a1e70 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:19:31.870218 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca1b1ce2-367c-4356-909d-fb63ca7e5a77 None admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:31.870317 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d42fe5c4-6caf-4933-8c83-f392057a1e70 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:31.870502 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca1b1ce2-367c-4356-909d-fb63ca7e5a77 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:31.870780 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca1b1ce2-367c-4356-909d-fb63ca7e5a77 None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:31.871467 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca1b1ce2-367c-4356-909d-fb63ca7e5a77 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:31.871669 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d42fe5c4-6caf-4933-8c83-f392057a1e70 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:31.873505 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca1b1ce2-367c-4356-909d-fb63ca7e5a77 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:31.891453 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 16/29] 10.222.0.90 () {56 vars in 1099 bytes} [Mon Aug 29 21:19:31 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 277 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:19:31.892467 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d42fe5c4-6caf-4933-8c83-f392057a1e70 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:31.992472 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:32.027338 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 14/30] 10.222.0.90 () {60 vars in 1131 bytes} [Mon Aug 29 21:19:31 2022] POST /identity/v3/roles => generated 219 bytes in 437 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:19:32.028768 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca1b1ce2-367c-4356-909d-fb63ca7e5a77 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:32.030985 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d42fe5c4-6caf-4933-8c83-f392057a1e70 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:32.074849 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-159370cb-3e8a-4978-81f1-3e35b1608b92 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:32.075514 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-159370cb-3e8a-4978-81f1-3e35b1608b92 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:32.075795 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-159370cb-3e8a-4978-81f1-3e35b1608b92 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:32.076067 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-159370cb-3e8a-4978-81f1-3e35b1608b92 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:32.102746 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-159370cb-3e8a-4978-81f1-3e35b1608b92 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:32.104447 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-159370cb-3e8a-4978-81f1-3e35b1608b92 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:32.136009 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-159370cb-3e8a-4978-81f1-3e35b1608b92 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:19:32.136009 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:19:32.136009 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:19:32.136009 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:19:32.136009 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:19:32.136009 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:19:32.136009 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:19:32.136009 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:19:32.136009 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:19:32.136009 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:19:32.136009 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:19:32.136009 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:19:32.136009 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 21:19:32.136009 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 21:19:32.136009 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 21:19:32.136009 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 21:19:32.136951 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:19:32.136951 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:19:32.136951 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:19:32.136951 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:19:32.136951 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:19:32.136951 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:19:32.136951 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:19:32.136951 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:19:32.136951 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:19:32.136951 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:19:32.136951 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:19:32.136951 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:19:32.136951 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:19:32.136951 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:19:32.136951 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:19:32.136951 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:19:32.136951 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 21:19:32.137767 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 21:19:32.137767 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:19:32.137767 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:19:32.139685 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 17/31] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:19:31 2022] GET /identity/v3/domains/Default => generated 87 bytes in 248 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:19:32.141346 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b68923d9-4569-49ec-9789-14073498ab6b None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:32.142035 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b68923d9-4569-49ec-9789-14073498ab6b None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:32.142605 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b68923d9-4569-49ec-9789-14073498ab6b None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:32.144793 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 18/32] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:19:32 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:19:32.146058 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b68923d9-4569-49ec-9789-14073498ab6b None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:32.159631 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca1b1ce2-367c-4356-909d-fb63ca7e5a77 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:32.235292 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28bd6eee-fc73-4928-9970-085b0b20bc48 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:32.236421 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28bd6eee-fc73-4928-9970-085b0b20bc48 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:32.236907 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28bd6eee-fc73-4928-9970-085b0b20bc48 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:32.237407 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28bd6eee-fc73-4928-9970-085b0b20bc48 None admin] PATH_INFO: `/v3/users/97a0b159abf44cd59fe5a2b7bde28adf` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:32.240615 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28bd6eee-fc73-4928-9970-085b0b20bc48 None admin] RBAC: Authorizing `identity:get_user(user_id=97a0b159abf44cd59fe5a2b7bde28adf)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:32.242562 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28bd6eee-fc73-4928-9970-085b0b20bc48 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:32.244517 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 15/33] 10.222.0.90 () {56 vars in 1177 bytes} [Mon Aug 29 21:19:32 2022] GET /identity/v3/users/97a0b159abf44cd59fe5a2b7bde28adf => generated 247 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:19:32.246130 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5cc7f37-d79f-4250-9427-709359d9e00f None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:32.246705 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5cc7f37-d79f-4250-9427-709359d9e00f None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:32.247184 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5cc7f37-d79f-4250-9427-709359d9e00f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:32.270377 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-c5cc7f37-d79f-4250-9427-709359d9e00f None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:19:32.272732 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-c5cc7f37-d79f-4250-9427-709359d9e00f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:19:32.274506 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b68923d9-4569-49ec-9789-14073498ab6b None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:32.293366 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:32.310182 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:32.312413 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b84e69a9-f910-4462-a4e8-68ab95c5acfd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:32.313402 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b84e69a9-f910-4462-a4e8-68ab95c5acfd None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:32.313402 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b84e69a9-f910-4462-a4e8-68ab95c5acfd None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:32.313815 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b84e69a9-f910-4462-a4e8-68ab95c5acfd None admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:32.314591 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b84e69a9-f910-4462-a4e8-68ab95c5acfd None admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:19:32.314935 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b84e69a9-f910-4462-a4e8-68ab95c5acfd None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:32.316421 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b84e69a9-f910-4462-a4e8-68ab95c5acfd None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:32.335581 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 19/34] 10.222.0.90 () {56 vars in 1109 bytes} [Mon Aug 29 21:19:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:19:32.337279 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b84e69a9-f910-4462-a4e8-68ab95c5acfd None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:32.364014 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 16/35] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:19:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 21:19:32.365922 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5cc7f37-d79f-4250-9427-709359d9e00f None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:32.448897 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b84e69a9-f910-4462-a4e8-68ab95c5acfd None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:32.484708 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d32506f-4904-430d-a5f0-ee69a811906e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:32.485542 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d32506f-4904-430d-a5f0-ee69a811906e None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:32.485920 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d32506f-4904-430d-a5f0-ee69a811906e None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:32.486728 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d32506f-4904-430d-a5f0-ee69a811906e None admin] PATH_INFO: `/v3/domains/default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:32.487531 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d32506f-4904-430d-a5f0-ee69a811906e None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:32.489061 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d32506f-4904-430d-a5f0-ee69a811906e None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:32.491354 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 20/36] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:19:32 2022] GET /identity/v3/domains/default => generated 199 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:19:32.498157 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d32506f-4904-430d-a5f0-ee69a811906e None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:32.578547 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5cc7f37-d79f-4250-9427-709359d9e00f None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:32.592054 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:32.648305 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e40c5398-3e30-48d1-ae5b-f6ea1471c88f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:32.648940 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e40c5398-3e30-48d1-ae5b-f6ea1471c88f None admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:32.649223 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e40c5398-3e30-48d1-ae5b-f6ea1471c88f None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:32.649498 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e40c5398-3e30-48d1-ae5b-f6ea1471c88f None admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:32.650181 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e40c5398-3e30-48d1-ae5b-f6ea1471c88f None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:32.651649 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e40c5398-3e30-48d1-ae5b-f6ea1471c88f None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:32.691496 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d32506f-4904-430d-a5f0-ee69a811906e None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:32.704499 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 17/37] 10.222.0.90 () {60 vars in 1141 bytes} [Mon Aug 29 21:19:32 2022] POST /identity/v3/projects => generated 301 bytes in 339 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:19:32.709764 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-693f5dc2-79cc-4087-bf32-aafc72cb99f7 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:32.710106 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-693f5dc2-79cc-4087-bf32-aafc72cb99f7 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:32.710349 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-693f5dc2-79cc-4087-bf32-aafc72cb99f7 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:32.712084 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 18/38] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:19:32 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:19:32.727385 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee530f1c-a267-419b-82af-e5d2d0c2d3e6 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:32.727679 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee530f1c-a267-419b-82af-e5d2d0c2d3e6 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:32.727951 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee530f1c-a267-419b-82af-e5d2d0c2d3e6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:32.737848 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40bd8067-3d32-4667-b3b9-f908e14007ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:32.738604 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40bd8067-3d32-4667-b3b9-f908e14007ba None admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:32.738881 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40bd8067-3d32-4667-b3b9-f908e14007ba None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:32.739181 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40bd8067-3d32-4667-b3b9-f908e14007ba None admin] PATH_INFO: `/v3/domains/default/users/97a0b159abf44cd59fe5a2b7bde28adf/roles/a6e562cdc96c4be691d638990f8403fd` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:32.741350 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40bd8067-3d32-4667-b3b9-f908e14007ba None admin] RBAC: Authorizing `identity:create_grant(domain_id=default, user_id=97a0b159abf44cd59fe5a2b7bde28adf, role_id=a6e562cdc96c4be691d638990f8403fd)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:32.742908 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40bd8067-3d32-4667-b3b9-f908e14007ba None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:32.763119 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-ee530f1c-a267-419b-82af-e5d2d0c2d3e6 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:19:32.764733 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-ee530f1c-a267-419b-82af-e5d2d0c2d3e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:19:32.803814 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:32.858230 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 19/39] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:19:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 363 bytes (3 switches on core 0) Aug 29 21:19:32.859432 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee530f1c-a267-419b-82af-e5d2d0c2d3e6 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:32.902584 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 21/40] 10.222.0.90 () {58 vars in 1361 bytes} [Mon Aug 29 21:19:32 2022] PUT /identity/v3/domains/default/users/97a0b159abf44cd59fe5a2b7bde28adf/roles/a6e562cdc96c4be691d638990f8403fd => generated 0 bytes in 405 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:19:32.961351 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a7344d4-f124-453f-abdc-98f739566a41 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:32.961642 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a7344d4-f124-453f-abdc-98f739566a41 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:32.961884 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a7344d4-f124-453f-abdc-98f739566a41 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:32.968223 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 22/41] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:19:32 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 29 21:19:32.974691 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbcff711-14f8-4eab-85cf-736d8cc378d5 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:32.975009 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbcff711-14f8-4eab-85cf-736d8cc378d5 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:32.975255 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbcff711-14f8-4eab-85cf-736d8cc378d5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:32.982708 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee530f1c-a267-419b-82af-e5d2d0c2d3e6 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:32.996773 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-dbcff711-14f8-4eab-85cf-736d8cc378d5 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:19:32.998385 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-dbcff711-14f8-4eab-85cf-736d8cc378d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:19:33.032772 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adfb2064-f224-4e14-8fa1-f350bbacc481 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:33.034305 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adfb2064-f224-4e14-8fa1-f350bbacc481 None admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:33.035095 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adfb2064-f224-4e14-8fa1-f350bbacc481 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:33.035581 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adfb2064-f224-4e14-8fa1-f350bbacc481 None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:33.038365 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adfb2064-f224-4e14-8fa1-f350bbacc481 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:33.040354 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adfb2064-f224-4e14-8fa1-f350bbacc481 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:33.063715 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 20/42] 10.222.0.90 () {60 vars in 1131 bytes} [Mon Aug 29 21:19:32 2022] POST /identity/v3/roles => generated 225 bytes in 205 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:19:33.090307 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 23/43] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:19:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 29 21:19:33.170406 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adfb2064-f224-4e14-8fa1-f350bbacc481 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:33.180733 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:33.289293 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adfb2064-f224-4e14-8fa1-f350bbacc481 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:33.326017 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46aa5879-9f6f-4115-8122-9d3a58560945 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:33.326647 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46aa5879-9f6f-4115-8122-9d3a58560945 None admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:33.326934 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46aa5879-9f6f-4115-8122-9d3a58560945 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:33.327214 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46aa5879-9f6f-4115-8122-9d3a58560945 None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:33.327905 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46aa5879-9f6f-4115-8122-9d3a58560945 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:33.329312 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46aa5879-9f6f-4115-8122-9d3a58560945 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:33.345739 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 21/44] 10.222.0.90 () {60 vars in 1131 bytes} [Mon Aug 29 21:19:33 2022] POST /identity/v3/roles => generated 223 bytes in 176 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:19:33.384222 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbcff711-14f8-4eab-85cf-736d8cc378d5 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:33.425473 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:33.501364 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbcff711-14f8-4eab-85cf-736d8cc378d5 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:33.642935 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a92e2b9f-010c-478e-8d2e-2e9c9054838a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:33.643496 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a92e2b9f-010c-478e-8d2e-2e9c9054838a None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:33.643782 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a92e2b9f-010c-478e-8d2e-2e9c9054838a None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:33.644060 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a92e2b9f-010c-478e-8d2e-2e9c9054838a None admin] PATH_INFO: `/v3/domains/default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:33.645183 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a92e2b9f-010c-478e-8d2e-2e9c9054838a None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:33.646885 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a92e2b9f-010c-478e-8d2e-2e9c9054838a None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:33.648487 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 24/45] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:19:33 2022] GET /identity/v3/domains/default => generated 199 bytes in 265 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:19:33.654188 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46aa5879-9f6f-4115-8122-9d3a58560945 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:33.719403 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:33.754814 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46aa5879-9f6f-4115-8122-9d3a58560945 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:33.789447 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e860ea0-8a01-4a35-883d-f7e366aefffa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:33.790190 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e860ea0-8a01-4a35-883d-f7e366aefffa None admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:33.790546 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e860ea0-8a01-4a35-883d-f7e366aefffa None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:33.790993 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e860ea0-8a01-4a35-883d-f7e366aefffa None admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:33.791653 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e860ea0-8a01-4a35-883d-f7e366aefffa None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:33.793122 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e860ea0-8a01-4a35-883d-f7e366aefffa None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:33.818433 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 22/46] 10.222.0.90 () {60 vars in 1141 bytes} [Mon Aug 29 21:19:33 2022] POST /identity/v3/projects => generated 312 bytes in 165 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:19:34.927311 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-641b5fad-5c39-4309-8d1d-b64d5cde890a None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:34.927848 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-641b5fad-5c39-4309-8d1d-b64d5cde890a None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:34.927923 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-641b5fad-5c39-4309-8d1d-b64d5cde890a None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:34.929250 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 25/47] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:19:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:19:34.934347 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ebd69f2-7cc2-4ade-9c59-d03f7902f5ad None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:34.934755 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ebd69f2-7cc2-4ade-9c59-d03f7902f5ad None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:34.934971 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ebd69f2-7cc2-4ade-9c59-d03f7902f5ad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:34.953572 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-3ebd69f2-7cc2-4ade-9c59-d03f7902f5ad None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:19:34.954811 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-3ebd69f2-7cc2-4ade-9c59-d03f7902f5ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:19:35.019917 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 23/48] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:19:34 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 29 21:19:35.293835 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-641b5fad-5c39-4309-8d1d-b64d5cde890a None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:35.398472 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-641b5fad-5c39-4309-8d1d-b64d5cde890a None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:35.400819 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:35.434042 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f068f59-76f5-472a-b6a9-7c1c96271e13 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:35.434682 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f068f59-76f5-472a-b6a9-7c1c96271e13 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:35.434964 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f068f59-76f5-472a-b6a9-7c1c96271e13 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:35.435237 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f068f59-76f5-472a-b6a9-7c1c96271e13 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:35.445930 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f068f59-76f5-472a-b6a9-7c1c96271e13 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 21:19:35.450699 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-3f068f59-76f5-472a-b6a9-7c1c96271e13 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 21:19:35.450699 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:19:35.450699 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:19:35.450699 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:19:35.450699 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:19:35.450699 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:19:35.450699 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:19:35.450699 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:19:35.450699 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:19:35.450699 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:19:35.450699 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:19:35.450699 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:19:35.450699 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 21:19:35.450699 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 21:19:35.450699 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 21:19:35.450699 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 21:19:35.451821 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 21:19:35.451821 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 21:19:35.451821 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 21:19:35.451821 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 21:19:35.451821 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:19:35.451821 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:19:35.451821 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:19:35.451821 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:19:35.451821 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:19:35.451821 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:19:35.451821 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:19:35.451821 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:19:35.451821 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:19:35.451821 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:19:35.451821 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:19:35.451821 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:19:35.452965 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:19:35.452965 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:19:35.452965 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:19:35.452965 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:19:35.452965 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 21:19:35.452965 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 21:19:35.452965 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 21:19:35.452965 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 21:19:35.452965 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 21:19:35.452965 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 21:19:35.452965 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 21:19:35.452965 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:19:35.452965 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 26/49] 10.222.0.90 () {56 vars in 1096 bytes} [Mon Aug 29 21:19:35 2022] GET /identity/v3/roles/admin => generated 83 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:19:35.457581 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ebd69f2-7cc2-4ade-9c59-d03f7902f5ad None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:35.506759 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:35.557928 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ebd69f2-7cc2-4ade-9c59-d03f7902f5ad None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:35.590397 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2c1fa9c-f794-46dc-a99f-9785b851b545 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:35.591012 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2c1fa9c-f794-46dc-a99f-9785b851b545 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:35.591290 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2c1fa9c-f794-46dc-a99f-9785b851b545 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:35.591562 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2c1fa9c-f794-46dc-a99f-9785b851b545 None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:35.592224 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2c1fa9c-f794-46dc-a99f-9785b851b545 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:19:35.592527 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2c1fa9c-f794-46dc-a99f-9785b851b545 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:35.593947 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2c1fa9c-f794-46dc-a99f-9785b851b545 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:35.606762 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 24/50] 10.222.0.90 () {56 vars in 1099 bytes} [Mon Aug 29 21:19:35 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:19:35.612816 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f068f59-76f5-472a-b6a9-7c1c96271e13 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:35.714045 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f068f59-76f5-472a-b6a9-7c1c96271e13 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:35.728619 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7a41ffc-3208-42f4-be45-affdaf62388f None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:35.728913 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7a41ffc-3208-42f4-be45-affdaf62388f None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:35.729197 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7a41ffc-3208-42f4-be45-affdaf62388f None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:35.730574 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 25/51] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:19:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 29 21:19:35.735331 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36192f40-9dd9-411c-bd2f-d9f26ea08153 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:35.735613 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36192f40-9dd9-411c-bd2f-d9f26ea08153 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:35.735884 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36192f40-9dd9-411c-bd2f-d9f26ea08153 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:35.749966 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-210ec24a-dc18-4fbb-ad2f-8894c29084bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:35.750608 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-210ec24a-dc18-4fbb-ad2f-8894c29084bc None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:35.750895 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-210ec24a-dc18-4fbb-ad2f-8894c29084bc None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:35.751180 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-210ec24a-dc18-4fbb-ad2f-8894c29084bc None admin] PATH_INFO: `/v3/users/97a0b159abf44cd59fe5a2b7bde28adf` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:35.752310 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-210ec24a-dc18-4fbb-ad2f-8894c29084bc None admin] RBAC: Authorizing `identity:get_user(user_id=97a0b159abf44cd59fe5a2b7bde28adf)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:35.753764 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-210ec24a-dc18-4fbb-ad2f-8894c29084bc None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:35.755532 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-36192f40-9dd9-411c-bd2f-d9f26ea08153 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:19:35.760382 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 27/52] 10.222.0.90 () {56 vars in 1177 bytes} [Mon Aug 29 21:19:35 2022] GET /identity/v3/users/97a0b159abf44cd59fe5a2b7bde28adf => generated 247 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:19:35.760527 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-36192f40-9dd9-411c-bd2f-d9f26ea08153 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:19:35.761344 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-210ec24a-dc18-4fbb-ad2f-8894c29084bc None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:35.792120 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:35.869447 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-210ec24a-dc18-4fbb-ad2f-8894c29084bc None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:35.905774 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bb68d9d-aac1-4490-95e5-bfb63e11302c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:35.906437 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb68d9d-aac1-4490-95e5-bfb63e11302c None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:35.906722 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb68d9d-aac1-4490-95e5-bfb63e11302c None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:35.907001 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb68d9d-aac1-4490-95e5-bfb63e11302c None admin] PATH_INFO: `/v3/domains/default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:35.908116 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bb68d9d-aac1-4490-95e5-bfb63e11302c None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:35.909747 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bb68d9d-aac1-4490-95e5-bfb63e11302c None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:35.911545 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 28/53] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:19:35 2022] GET /identity/v3/domains/default => generated 199 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:19:35.917225 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bb68d9d-aac1-4490-95e5-bfb63e11302c None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:35.929996 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 26/54] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:19:35 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 195 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 29 21:19:36.012114 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:36.015430 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bb68d9d-aac1-4490-95e5-bfb63e11302c None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:36.050082 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8b057e8-1b04-471c-a05e-acc7f15a7e16 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:36.050697 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8b057e8-1b04-471c-a05e-acc7f15a7e16 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:36.050983 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8b057e8-1b04-471c-a05e-acc7f15a7e16 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:36.051260 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8b057e8-1b04-471c-a05e-acc7f15a7e16 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:36.052019 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8b057e8-1b04-471c-a05e-acc7f15a7e16 None admin] RBAC: Adding query filter params (role.id=a6e562cdc96c4be691d638990f8403fd, scope.domain.id=default, user.id=97a0b159abf44cd59fe5a2b7bde28adf) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:19:36.052386 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8b057e8-1b04-471c-a05e-acc7f15a7e16 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:36.053875 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8b057e8-1b04-471c-a05e-acc7f15a7e16 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:36.059592 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 29/55] 10.222.0.90 () {56 vars in 1322 bytes} [Mon Aug 29 21:19:35 2022] GET /identity/v3/role_assignments?user.id=97a0b159abf44cd59fe5a2b7bde28adf&scope.domain.id=default&role.id=a6e562cdc96c4be691d638990f8403fd => generated 530 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:19:36.200686 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36192f40-9dd9-411c-bd2f-d9f26ea08153 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:36.243051 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:36.302744 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36192f40-9dd9-411c-bd2f-d9f26ea08153 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:36.335941 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dae7211c-a34c-416d-b38d-9ed2668a793e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:36.336577 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dae7211c-a34c-416d-b38d-9ed2668a793e None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:36.336864 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dae7211c-a34c-416d-b38d-9ed2668a793e None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:36.337144 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dae7211c-a34c-416d-b38d-9ed2668a793e None admin] PATH_INFO: `/v3/domains/default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:36.338287 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dae7211c-a34c-416d-b38d-9ed2668a793e None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:36.339749 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dae7211c-a34c-416d-b38d-9ed2668a793e None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:36.341331 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 27/56] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:19:36 2022] GET /identity/v3/domains/default => generated 199 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:19:36.347329 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8b057e8-1b04-471c-a05e-acc7f15a7e16 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:36.456422 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8b057e8-1b04-471c-a05e-acc7f15a7e16 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:36.490172 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34bf2ff1-a208-48e9-a546-ab65d19df3dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:36.490777 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34bf2ff1-a208-48e9-a546-ab65d19df3dc None admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:36.491058 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34bf2ff1-a208-48e9-a546-ab65d19df3dc None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:36.491339 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34bf2ff1-a208-48e9-a546-ab65d19df3dc None admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:36.491983 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34bf2ff1-a208-48e9-a546-ab65d19df3dc None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:36.493426 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34bf2ff1-a208-48e9-a546-ab65d19df3dc None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:36.522843 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 30/57] 10.222.0.90 () {60 vars in 1140 bytes} [Mon Aug 29 21:19:36 2022] POST /identity/v3/projects => generated 298 bytes in 176 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:19:38.451871 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96cfef75-b0ff-411e-8d23-ef35d93e66e0 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:38.452390 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96cfef75-b0ff-411e-8d23-ef35d93e66e0 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:38.452475 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96cfef75-b0ff-411e-8d23-ef35d93e66e0 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:38.453756 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 28/58] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:19:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:19:38.459561 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e06610-7f29-435c-b3bb-fd47b2dedf85 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:38.459976 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e06610-7f29-435c-b3bb-fd47b2dedf85 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:38.460125 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e06610-7f29-435c-b3bb-fd47b2dedf85 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:38.478872 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-e0e06610-7f29-435c-b3bb-fd47b2dedf85 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:19:38.480084 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-e0e06610-7f29-435c-b3bb-fd47b2dedf85 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:19:38.490583 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:38.547240 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 31/59] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:19:38 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 21:19:38.814745 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96cfef75-b0ff-411e-8d23-ef35d93e66e0 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:38.916906 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96cfef75-b0ff-411e-8d23-ef35d93e66e0 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:38.925481 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:38.951083 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78ca65a5-cb50-4321-a615-b6f1a1bacf0c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:38.951723 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78ca65a5-cb50-4321-a615-b6f1a1bacf0c None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:38.952010 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78ca65a5-cb50-4321-a615-b6f1a1bacf0c None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:38.952287 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78ca65a5-cb50-4321-a615-b6f1a1bacf0c None admin] PATH_INFO: `/v3/domains/default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:38.953413 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78ca65a5-cb50-4321-a615-b6f1a1bacf0c None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:38.954898 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78ca65a5-cb50-4321-a615-b6f1a1bacf0c None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:38.956473 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 29/60] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:19:38 2022] GET /identity/v3/domains/default => generated 199 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:19:38.962852 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0e06610-7f29-435c-b3bb-fd47b2dedf85 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:39.060078 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0e06610-7f29-435c-b3bb-fd47b2dedf85 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:39.093157 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2307e1d1-70d1-477a-830f-039917271951 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:39.093706 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2307e1d1-70d1-477a-830f-039917271951 None admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:39.094020 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2307e1d1-70d1-477a-830f-039917271951 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:39.094300 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2307e1d1-70d1-477a-830f-039917271951 None admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:39.094966 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2307e1d1-70d1-477a-830f-039917271951 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:39.096669 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2307e1d1-70d1-477a-830f-039917271951 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:39.124228 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 32/61] 10.222.0.90 () {60 vars in 1132 bytes} [Mon Aug 29 21:19:38 2022] POST /identity/v3/users => generated 275 bytes in 162 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 21:19:42.186557 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0eed079-8de8-4cb9-af94-aefd52fc5305 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:42.187529 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0eed079-8de8-4cb9-af94-aefd52fc5305 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:42.188037 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0eed079-8de8-4cb9-af94-aefd52fc5305 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:42.189607 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 30/62] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:19:42 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:19:42.202544 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dd8496a-b8cf-4af8-82c1-d3a1e4162312 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:42.203509 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dd8496a-b8cf-4af8-82c1-d3a1e4162312 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:42.204000 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dd8496a-b8cf-4af8-82c1-d3a1e4162312 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:42.213225 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-561354fd-be2e-441f-be0a-39051ae6e4cb None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:42.213420 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:42.213976 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-561354fd-be2e-441f-be0a-39051ae6e4cb None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:42.214453 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-561354fd-be2e-441f-be0a-39051ae6e4cb None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:42.216583 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 31/63] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:19:42 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 29 21:19:42.222545 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b3c2efd-3068-467a-9dfb-1e866b32d91e None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:42.223096 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b3c2efd-3068-467a-9dfb-1e866b32d91e None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:42.223882 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b3c2efd-3068-467a-9dfb-1e866b32d91e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:42.231268 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-7dd8496a-b8cf-4af8-82c1-d3a1e4162312 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:19:42.232785 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-7dd8496a-b8cf-4af8-82c1-d3a1e4162312 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:19:42.255581 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-0b3c2efd-3068-467a-9dfb-1e866b32d91e None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:19:42.257038 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-0b3c2efd-3068-467a-9dfb-1e866b32d91e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:19:42.314455 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 33/64] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:19:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 21:19:42.386330 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 32/65] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:19:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 29 21:19:42.589526 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dd8496a-b8cf-4af8-82c1-d3a1e4162312 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:42.676635 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fb76806-28ac-4927-9dee-2a562bb2f201 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:42.677442 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fb76806-28ac-4927-9dee-2a562bb2f201 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:42.677695 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fb76806-28ac-4927-9dee-2a562bb2f201 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:42.680023 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 33/66] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:19:42 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 29 21:19:42.687856 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15f4bbf1-63e4-4a7d-b989-522afe513107 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:42.688386 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15f4bbf1-63e4-4a7d-b989-522afe513107 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:42.689025 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15f4bbf1-63e4-4a7d-b989-522afe513107 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:42.691212 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 34/67] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:19:42 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 29 21:19:42.692991 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d114aa1-f852-49e5-bbd7-62d0c384571b None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:42.693508 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d114aa1-f852-49e5-bbd7-62d0c384571b None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:42.694025 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d114aa1-f852-49e5-bbd7-62d0c384571b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:42.711599 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dd8496a-b8cf-4af8-82c1-d3a1e4162312 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:42.713603 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-8d114aa1-f852-49e5-bbd7-62d0c384571b None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:19:42.714833 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-8d114aa1-f852-49e5-bbd7-62d0c384571b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:19:42.748203 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:42.754975 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be7c793e-011d-46ae-bd90-d300503bc9a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:42.755636 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be7c793e-011d-46ae-bd90-d300503bc9a8 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:42.755922 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be7c793e-011d-46ae-bd90-d300503bc9a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:42.756198 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be7c793e-011d-46ae-bd90-d300503bc9a8 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:42.769344 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be7c793e-011d-46ae-bd90-d300503bc9a8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 21:19:42.775091 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-be7c793e-011d-46ae-bd90-d300503bc9a8 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 29 21:19:42.775091 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:19:42.775091 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:19:42.775091 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:19:42.775091 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:19:42.775091 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:19:42.775091 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:19:42.775091 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:19:42.775091 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:19:42.775091 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:19:42.775091 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:19:42.775091 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:19:42.775091 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 21:19:42.775091 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 21:19:42.775091 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 21:19:42.775091 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 21:19:42.775975 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 21:19:42.775975 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 21:19:42.775975 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 21:19:42.775975 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 21:19:42.775975 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:19:42.775975 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:19:42.775975 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:19:42.775975 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:19:42.775975 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:19:42.775975 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:19:42.775975 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:19:42.775975 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:19:42.775975 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:19:42.775975 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:19:42.775975 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:19:42.775975 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:19:42.776757 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:19:42.776757 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:19:42.776757 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:19:42.776757 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:19:42.776757 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 21:19:42.776757 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 21:19:42.776757 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 21:19:42.776757 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 21:19:42.776757 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 21:19:42.776757 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 21:19:42.776757 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 29 21:19:42.776757 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:19:42.776757 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 34/68] 10.222.0.90 () {56 vars in 1099 bytes} [Mon Aug 29 21:19:42 2022] GET /identity/v3/roles/member => generated 84 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:19:42.777390 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be7c793e-011d-46ae-bd90-d300503bc9a8 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:42.795141 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 35/69] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:19:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 21:19:42.798012 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b20623-cf96-4abc-8344-3035ed3a5a7c None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:42.798012 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b20623-cf96-4abc-8344-3035ed3a5a7c None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:42.798012 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b20623-cf96-4abc-8344-3035ed3a5a7c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:42.820558 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-05b20623-cf96-4abc-8344-3035ed3a5a7c None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:19:42.821761 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-05b20623-cf96-4abc-8344-3035ed3a5a7c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:19:42.869037 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:42.894497 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be7c793e-011d-46ae-bd90-d300503bc9a8 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:42.918992 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 36/70] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:19:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 21:19:42.919824 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05b20623-cf96-4abc-8344-3035ed3a5a7c None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:42.932613 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-700e45e5-286b-4256-8e94-4ca9756303f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:42.933263 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-700e45e5-286b-4256-8e94-4ca9756303f7 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:42.933604 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-700e45e5-286b-4256-8e94-4ca9756303f7 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:42.933916 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-700e45e5-286b-4256-8e94-4ca9756303f7 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:42.944913 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-700e45e5-286b-4256-8e94-4ca9756303f7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 21:19:42.950057 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-700e45e5-286b-4256-8e94-4ca9756303f7 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 21:19:42.950057 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:19:42.950057 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:19:42.950057 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:19:42.950057 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:19:42.950057 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:19:42.950057 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:19:42.950057 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:19:42.950057 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:19:42.950057 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:19:42.950057 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:19:42.950057 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:19:42.950057 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 21:19:42.950057 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 21:19:42.950057 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 21:19:42.950057 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 21:19:42.950919 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 21:19:42.950919 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 21:19:42.950919 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 21:19:42.950919 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 21:19:42.950919 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:19:42.950919 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:19:42.950919 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:19:42.950919 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:19:42.950919 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:19:42.950919 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:19:42.950919 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:19:42.950919 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:19:42.950919 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:19:42.950919 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:19:42.950919 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:19:42.950919 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:19:42.951742 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:19:42.951742 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:19:42.951742 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:19:42.951742 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:19:42.951742 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 21:19:42.951742 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 21:19:42.951742 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 21:19:42.951742 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 21:19:42.951742 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 21:19:42.951742 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 21:19:42.951742 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 21:19:42.951742 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:19:42.953955 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 35/71] 10.222.0.90 () {56 vars in 1096 bytes} [Mon Aug 29 21:19:42 2022] GET /identity/v3/roles/admin => generated 83 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:19:42.955313 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06e9fc4b-87d0-4e3c-9479-16472551fe7e None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:42.955798 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06e9fc4b-87d0-4e3c-9479-16472551fe7e None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:42.956278 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06e9fc4b-87d0-4e3c-9479-16472551fe7e None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:42.958834 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 36/72] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:19:42 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:19:42.969817 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc3f18e7-0db7-41ca-82b7-4ef3d21b239e None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:42.969817 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc3f18e7-0db7-41ca-82b7-4ef3d21b239e None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:42.969817 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc3f18e7-0db7-41ca-82b7-4ef3d21b239e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:42.989383 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-fc3f18e7-0db7-41ca-82b7-4ef3d21b239e None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:19:42.991845 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-fc3f18e7-0db7-41ca-82b7-4ef3d21b239e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:19:43.051822 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05b20623-cf96-4abc-8344-3035ed3a5a7c None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:43.052869 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:43.076460 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 37/73] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:19:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 21:19:43.077555 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc3f18e7-0db7-41ca-82b7-4ef3d21b239e None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:43.112474 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5230aef1-e785-4bb1-9033-066f0219c879 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:43.113021 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5230aef1-e785-4bb1-9033-066f0219c879 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:43.113307 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5230aef1-e785-4bb1-9033-066f0219c879 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:43.113581 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5230aef1-e785-4bb1-9033-066f0219c879 None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:43.114324 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5230aef1-e785-4bb1-9033-066f0219c879 None admin] RBAC: Adding query filter params (name=member) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:19:43.114631 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5230aef1-e785-4bb1-9033-066f0219c879 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:43.116039 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5230aef1-e785-4bb1-9033-066f0219c879 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:43.125765 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 37/74] 10.222.0.90 () {56 vars in 1101 bytes} [Mon Aug 29 21:19:42 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:19:43.126603 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5230aef1-e785-4bb1-9033-066f0219c879 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:43.162317 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:43.213070 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc3f18e7-0db7-41ca-82b7-4ef3d21b239e None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:43.243746 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5230aef1-e785-4bb1-9033-066f0219c879 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:43.250000 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0047f6a6-b11e-429e-bd01-8dee7535e49d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:43.250619 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0047f6a6-b11e-429e-bd01-8dee7535e49d None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:43.250909 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0047f6a6-b11e-429e-bd01-8dee7535e49d None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:43.251180 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0047f6a6-b11e-429e-bd01-8dee7535e49d None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:43.252006 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0047f6a6-b11e-429e-bd01-8dee7535e49d None admin] RBAC: Adding query filter params (name=admin) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:19:43.252313 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0047f6a6-b11e-429e-bd01-8dee7535e49d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:43.253701 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0047f6a6-b11e-429e-bd01-8dee7535e49d None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:43.261973 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 38/75] 10.222.0.90 () {56 vars in 1099 bytes} [Mon Aug 29 21:19:43 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:19:43.266967 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0047f6a6-b11e-429e-bd01-8dee7535e49d None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:43.279546 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05dea800-d9f9-4de5-bf39-c7a3edff4007 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:43.280192 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05dea800-d9f9-4de5-bf39-c7a3edff4007 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:43.280486 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05dea800-d9f9-4de5-bf39-c7a3edff4007 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:43.280768 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05dea800-d9f9-4de5-bf39-c7a3edff4007 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:43.294519 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05dea800-d9f9-4de5-bf39-c7a3edff4007 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 21:19:43.300829 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-05dea800-d9f9-4de5-bf39-c7a3edff4007 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 29 21:19:43.300829 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:19:43.300829 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:19:43.300829 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:19:43.300829 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:19:43.300829 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:19:43.300829 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:19:43.300829 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:19:43.300829 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:19:43.300829 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:19:43.300829 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:19:43.300829 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:19:43.300829 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 21:19:43.300829 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 21:19:43.300829 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 21:19:43.300829 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 21:19:43.302385 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 21:19:43.302385 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 21:19:43.302385 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 21:19:43.302385 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 21:19:43.302385 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:19:43.302385 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:19:43.302385 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:19:43.302385 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:19:43.302385 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:19:43.302385 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:19:43.302385 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:19:43.302385 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:19:43.302385 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:19:43.302385 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:19:43.302385 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:19:43.302385 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:19:43.303261 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:19:43.303261 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:19:43.303261 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:19:43.303261 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:19:43.303261 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 21:19:43.303261 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 21:19:43.303261 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 21:19:43.303261 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 21:19:43.303261 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 21:19:43.303261 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 21:19:43.303261 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 29 21:19:43.303261 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:19:43.303261 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 38/76] 10.222.0.90 () {56 vars in 1114 bytes} [Mon Aug 29 21:19:43 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:19:43.303925 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05dea800-d9f9-4de5-bf39-c7a3edff4007 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:43.322562 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:43.381918 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0047f6a6-b11e-429e-bd01-8dee7535e49d None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:43.397918 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:43.411388 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05dea800-d9f9-4de5-bf39-c7a3edff4007 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:43.416157 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a24c9ab-5b05-4f00-bf88-071bc4c5d751 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:43.416803 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a24c9ab-5b05-4f00-bf88-071bc4c5d751 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:43.417090 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a24c9ab-5b05-4f00-bf88-071bc4c5d751 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:43.417371 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a24c9ab-5b05-4f00-bf88-071bc4c5d751 None admin] PATH_INFO: `/v3/users/94fd78dbbf6e44308bd6e72458d6361f` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:43.431213 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a24c9ab-5b05-4f00-bf88-071bc4c5d751 None admin] RBAC: Authorizing `identity:get_user(user_id=94fd78dbbf6e44308bd6e72458d6361f)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:43.432671 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a24c9ab-5b05-4f00-bf88-071bc4c5d751 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:43.434380 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 39/77] 10.222.0.90 () {56 vars in 1177 bytes} [Mon Aug 29 21:19:43 2022] GET /identity/v3/users/94fd78dbbf6e44308bd6e72458d6361f => generated 275 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:19:43.435151 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a24c9ab-5b05-4f00-bf88-071bc4c5d751 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:43.445752 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23106be0-3c39-4c29-b2c3-087c1422926d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:43.446419 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23106be0-3c39-4c29-b2c3-087c1422926d None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:43.446707 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23106be0-3c39-4c29-b2c3-087c1422926d None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:43.446987 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23106be0-3c39-4c29-b2c3-087c1422926d None admin] PATH_INFO: `/v3/roles/member` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:43.457040 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23106be0-3c39-4c29-b2c3-087c1422926d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 21:19:43.462934 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-23106be0-3c39-4c29-b2c3-087c1422926d None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 29 21:19:43.462934 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:19:43.462934 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:19:43.462934 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:19:43.462934 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:19:43.462934 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:19:43.462934 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:19:43.462934 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:19:43.462934 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:19:43.462934 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:19:43.462934 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:19:43.462934 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:19:43.462934 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 21:19:43.462934 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 21:19:43.462934 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 21:19:43.462934 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 21:19:43.463834 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 21:19:43.463834 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 21:19:43.463834 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 21:19:43.463834 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 21:19:43.463834 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:19:43.463834 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:19:43.463834 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:19:43.463834 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:19:43.463834 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:19:43.463834 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:19:43.463834 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:19:43.463834 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:19:43.463834 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:19:43.463834 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:19:43.463834 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:19:43.463834 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:19:43.464620 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:19:43.464620 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:19:43.464620 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:19:43.464620 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:19:43.464620 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 21:19:43.464620 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 21:19:43.464620 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 21:19:43.464620 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 21:19:43.464620 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 21:19:43.464620 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 21:19:43.464620 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 29 21:19:43.464620 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:19:43.464620 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 39/78] 10.222.0.90 () {56 vars in 1099 bytes} [Mon Aug 29 21:19:43 2022] GET /identity/v3/roles/member => generated 84 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:19:43.465271 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23106be0-3c39-4c29-b2c3-087c1422926d None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:43.537761 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a24c9ab-5b05-4f00-bf88-071bc4c5d751 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:43.539316 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:43.568177 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23106be0-3c39-4c29-b2c3-087c1422926d None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:43.573957 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-477b3a4b-1173-48e3-9d38-15d9173ebeb2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:43.574594 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-477b3a4b-1173-48e3-9d38-15d9173ebeb2 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:43.574886 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-477b3a4b-1173-48e3-9d38-15d9173ebeb2 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:43.575163 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-477b3a4b-1173-48e3-9d38-15d9173ebeb2 None admin] PATH_INFO: `/v3/users/97a0b159abf44cd59fe5a2b7bde28adf` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:43.576297 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-477b3a4b-1173-48e3-9d38-15d9173ebeb2 None admin] RBAC: Authorizing `identity:get_user(user_id=97a0b159abf44cd59fe5a2b7bde28adf)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:43.577758 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-477b3a4b-1173-48e3-9d38-15d9173ebeb2 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:43.579386 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 40/79] 10.222.0.90 () {56 vars in 1177 bytes} [Mon Aug 29 21:19:43 2022] GET /identity/v3/users/97a0b159abf44cd59fe5a2b7bde28adf => generated 247 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:19:43.580150 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-477b3a4b-1173-48e3-9d38-15d9173ebeb2 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:43.601718 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1e8bc88-9592-421a-9332-cdabadac0748 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:43.602463 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e8bc88-9592-421a-9332-cdabadac0748 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:43.602752 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e8bc88-9592-421a-9332-cdabadac0748 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:43.603048 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e8bc88-9592-421a-9332-cdabadac0748 None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:43.603746 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1e8bc88-9592-421a-9332-cdabadac0748 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:19:43.604051 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1e8bc88-9592-421a-9332-cdabadac0748 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:43.605448 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1e8bc88-9592-421a-9332-cdabadac0748 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:43.613413 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 40/80] 10.222.0.90 () {56 vars in 1111 bytes} [Mon Aug 29 21:19:43 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:19:43.618312 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1e8bc88-9592-421a-9332-cdabadac0748 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:43.626903 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:43.687966 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-477b3a4b-1173-48e3-9d38-15d9173ebeb2 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:43.716175 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1e8bc88-9592-421a-9332-cdabadac0748 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:43.722730 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b77c531b-d6f4-4b29-9ffc-90ca2ea3af1d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:43.723401 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b77c531b-d6f4-4b29-9ffc-90ca2ea3af1d None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:43.723813 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b77c531b-d6f4-4b29-9ffc-90ca2ea3af1d None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:43.724199 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b77c531b-d6f4-4b29-9ffc-90ca2ea3af1d None admin] PATH_INFO: `/v3/domains/default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:43.725346 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b77c531b-d6f4-4b29-9ffc-90ca2ea3af1d None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:43.726830 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b77c531b-d6f4-4b29-9ffc-90ca2ea3af1d None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:43.728419 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 41/81] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:19:43 2022] GET /identity/v3/domains/default => generated 199 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:19:43.729181 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b77c531b-d6f4-4b29-9ffc-90ca2ea3af1d None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:43.751756 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a78e698a-f731-4f05-874c-7ec2cf7f6291 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:43.752381 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a78e698a-f731-4f05-874c-7ec2cf7f6291 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:43.752667 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a78e698a-f731-4f05-874c-7ec2cf7f6291 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:43.752951 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a78e698a-f731-4f05-874c-7ec2cf7f6291 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:43.772241 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:43.820676 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a78e698a-f731-4f05-874c-7ec2cf7f6291 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:43.822177 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a78e698a-f731-4f05-874c-7ec2cf7f6291 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:43.834188 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:43.835241 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b77c531b-d6f4-4b29-9ffc-90ca2ea3af1d None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:43.872389 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b6c90bb-e4a4-400f-a031-56b2b629cdb6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:43.873012 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b6c90bb-e4a4-400f-a031-56b2b629cdb6 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:43.873295 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b6c90bb-e4a4-400f-a031-56b2b629cdb6 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:43.873572 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b6c90bb-e4a4-400f-a031-56b2b629cdb6 None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:43.874273 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b6c90bb-e4a4-400f-a031-56b2b629cdb6 None admin] RBAC: Adding query filter params (name=member) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:19:43.874583 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b6c90bb-e4a4-400f-a031-56b2b629cdb6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:43.875961 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b6c90bb-e4a4-400f-a031-56b2b629cdb6 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:43.885261 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 42/82] 10.222.0.90 () {56 vars in 1101 bytes} [Mon Aug 29 21:19:43 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:19:43.885261 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b6c90bb-e4a4-400f-a031-56b2b629cdb6 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:43.926279 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 41/83] 10.222.0.90 () {58 vars in 1282 bytes} [Mon Aug 29 21:19:43 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 312 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 29 21:19:43.927684 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a78e698a-f731-4f05-874c-7ec2cf7f6291 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:43.988739 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:43.992616 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b6c90bb-e4a4-400f-a031-56b2b629cdb6 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:44.026376 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edb5409a-e980-4821-82cd-f77938bc625e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:44.027054 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edb5409a-e980-4821-82cd-f77938bc625e None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:44.027338 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edb5409a-e980-4821-82cd-f77938bc625e None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:44.027611 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edb5409a-e980-4821-82cd-f77938bc625e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:44.028361 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a78e698a-f731-4f05-874c-7ec2cf7f6291 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:44.039808 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:44.064396 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd60799b-9d10-4890-9f50-2e59ec1943b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:44.065036 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd60799b-9d10-4890-9f50-2e59ec1943b5 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:44.065317 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd60799b-9d10-4890-9f50-2e59ec1943b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:44.065592 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd60799b-9d10-4890-9f50-2e59ec1943b5 None admin] PATH_INFO: `/v3/users/94fd78dbbf6e44308bd6e72458d6361f` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:44.066759 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd60799b-9d10-4890-9f50-2e59ec1943b5 None admin] RBAC: Authorizing `identity:get_user(user_id=94fd78dbbf6e44308bd6e72458d6361f)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:44.068364 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd60799b-9d10-4890-9f50-2e59ec1943b5 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:44.069980 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 42/84] 10.222.0.90 () {56 vars in 1177 bytes} [Mon Aug 29 21:19:43 2022] GET /identity/v3/users/94fd78dbbf6e44308bd6e72458d6361f => generated 275 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:19:44.070721 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd60799b-9d10-4890-9f50-2e59ec1943b5 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:44.101424 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edb5409a-e980-4821-82cd-f77938bc625e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:44.102948 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edb5409a-e980-4821-82cd-f77938bc625e None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:44.169068 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd60799b-9d10-4890-9f50-2e59ec1943b5 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:44.198458 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:44.202806 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 43/85] 10.222.0.90 () {58 vars in 1282 bytes} [Mon Aug 29 21:19:43 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 318 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 29 21:19:44.203780 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edb5409a-e980-4821-82cd-f77938bc625e None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:44.204152 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d9cb7ba-69e0-4156-b6e7-63403acc52dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:44.204952 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d9cb7ba-69e0-4156-b6e7-63403acc52dd None admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:44.205381 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d9cb7ba-69e0-4156-b6e7-63403acc52dd None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:44.205912 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d9cb7ba-69e0-4156-b6e7-63403acc52dd None admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:44.206507 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d9cb7ba-69e0-4156-b6e7-63403acc52dd None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:44.208491 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d9cb7ba-69e0-4156-b6e7-63403acc52dd None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:44.237259 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 43/86] 10.222.0.90 () {60 vars in 1132 bytes} [Mon Aug 29 21:19:44 2022] POST /identity/v3/users => generated 289 bytes in 167 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:19:44.238123 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d9cb7ba-69e0-4156-b6e7-63403acc52dd None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:44.288924 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:44.322518 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edb5409a-e980-4821-82cd-f77938bc625e None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:44.362493 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6e874ac-553e-43ea-b384-86745fb43ea7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:44.363160 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6e874ac-553e-43ea-b384-86745fb43ea7 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:44.363477 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6e874ac-553e-43ea-b384-86745fb43ea7 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:44.363725 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6e874ac-553e-43ea-b384-86745fb43ea7 None admin] PATH_INFO: `/v3/users/94fd78dbbf6e44308bd6e72458d6361f` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:44.365356 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6e874ac-553e-43ea-b384-86745fb43ea7 None admin] RBAC: Authorizing `identity:get_user(user_id=94fd78dbbf6e44308bd6e72458d6361f)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:44.366895 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6e874ac-553e-43ea-b384-86745fb43ea7 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:44.368904 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 44/87] 10.222.0.90 () {56 vars in 1177 bytes} [Mon Aug 29 21:19:44 2022] GET /identity/v3/users/94fd78dbbf6e44308bd6e72458d6361f => generated 275 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:19:44.369841 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6e874ac-553e-43ea-b384-86745fb43ea7 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:44.375696 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d9cb7ba-69e0-4156-b6e7-63403acc52dd None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:44.416337 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57590cce-ff4e-48bd-a539-83d58837a2e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:44.416987 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57590cce-ff4e-48bd-a539-83d58837a2e6 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:44.417269 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57590cce-ff4e-48bd-a539-83d58837a2e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:44.417543 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57590cce-ff4e-48bd-a539-83d58837a2e6 None admin] PATH_INFO: `/v3/projects/381cc34791e147329798cd454f99f622` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:44.418712 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57590cce-ff4e-48bd-a539-83d58837a2e6 None admin] RBAC: Authorizing `identity:get_project(project_id=381cc34791e147329798cd454f99f622)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:44.420394 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57590cce-ff4e-48bd-a539-83d58837a2e6 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:44.422131 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 44/88] 10.222.0.90 () {56 vars in 1186 bytes} [Mon Aug 29 21:19:44 2022] GET /identity/v3/projects/381cc34791e147329798cd454f99f622 => generated 298 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:19:44.423000 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57590cce-ff4e-48bd-a539-83d58837a2e6 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:44.450804 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:44.498553 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6e874ac-553e-43ea-b384-86745fb43ea7 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:44.543168 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eeb2241-8923-441b-8f70-13dac640988a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:44.543720 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eeb2241-8923-441b-8f70-13dac640988a None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:44.544011 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eeb2241-8923-441b-8f70-13dac640988a None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:44.544292 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eeb2241-8923-441b-8f70-13dac640988a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:44.552462 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57590cce-ff4e-48bd-a539-83d58837a2e6 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:44.553041 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:44.600175 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef77766a-3b48-4750-9897-37183ab1c30e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:44.601353 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef77766a-3b48-4750-9897-37183ab1c30e None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:44.601647 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef77766a-3b48-4750-9897-37183ab1c30e None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:44.602322 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef77766a-3b48-4750-9897-37183ab1c30e None admin] PATH_INFO: `/v3/projects/381cc34791e147329798cd454f99f622` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:44.603790 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef77766a-3b48-4750-9897-37183ab1c30e None admin] RBAC: Authorizing `identity:get_project(project_id=381cc34791e147329798cd454f99f622)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:44.605722 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef77766a-3b48-4750-9897-37183ab1c30e None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:44.608391 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 45/89] 10.222.0.90 () {56 vars in 1186 bytes} [Mon Aug 29 21:19:44 2022] GET /identity/v3/projects/381cc34791e147329798cd454f99f622 => generated 298 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:19:44.609197 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef77766a-3b48-4750-9897-37183ab1c30e None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:44.632110 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2eeb2241-8923-441b-8f70-13dac640988a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:44.634083 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2eeb2241-8923-441b-8f70-13dac640988a None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:44.726002 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:44.751933 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef77766a-3b48-4750-9897-37183ab1c30e None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:44.772422 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 45/90] 10.222.0.90 () {58 vars in 1282 bytes} [Mon Aug 29 21:19:44 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 403 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 29 21:19:44.773590 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eeb2241-8923-441b-8f70-13dac640988a None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:44.803408 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3b0bb41-5182-480b-9b29-f31250655751 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:44.804206 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3b0bb41-5182-480b-9b29-f31250655751 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:44.804494 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3b0bb41-5182-480b-9b29-f31250655751 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:44.805205 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3b0bb41-5182-480b-9b29-f31250655751 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:44.864850 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:44.890625 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eeb2241-8923-441b-8f70-13dac640988a None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:44.900122 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3b0bb41-5182-480b-9b29-f31250655751 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:44.901747 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3b0bb41-5182-480b-9b29-f31250655751 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:44.931982 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fb13c10-ad97-49a7-954b-57bb258b274c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:44.933139 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb13c10-ad97-49a7-954b-57bb258b274c None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:44.934053 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb13c10-ad97-49a7-954b-57bb258b274c None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:44.934513 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb13c10-ad97-49a7-954b-57bb258b274c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:44.935712 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fb13c10-ad97-49a7-954b-57bb258b274c None admin] RBAC: Adding query filter params (role.id=02e242d5520740d4b77a5f4008c7eb57, scope.project.id=381cc34791e147329798cd454f99f622, user.id=94fd78dbbf6e44308bd6e72458d6361f) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:19:44.936203 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fb13c10-ad97-49a7-954b-57bb258b274c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:44.938972 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fb13c10-ad97-49a7-954b-57bb258b274c None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:44.947072 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 46/91] 10.222.0.90 () {56 vars in 1374 bytes} [Mon Aug 29 21:19:44 2022] GET /identity/v3/role_assignments?user.id=94fd78dbbf6e44308bd6e72458d6361f&scope.project.id=381cc34791e147329798cd454f99f622&role.id=02e242d5520740d4b77a5f4008c7eb57 => generated 259 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:19:44.954680 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fb13c10-ad97-49a7-954b-57bb258b274c None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:45.033735 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:45.066652 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 46/92] 10.222.0.90 () {58 vars in 1282 bytes} [Mon Aug 29 21:19:44 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 458 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 29 21:19:45.073105 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3b0bb41-5182-480b-9b29-f31250655751 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:45.151902 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:45.174928 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fb13c10-ad97-49a7-954b-57bb258b274c None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:45.187623 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3b0bb41-5182-480b-9b29-f31250655751 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:45.210846 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-670da3a3-6bb0-48c1-983a-003fc9290167 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:45.211459 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-670da3a3-6bb0-48c1-983a-003fc9290167 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:45.211741 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-670da3a3-6bb0-48c1-983a-003fc9290167 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:45.212015 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-670da3a3-6bb0-48c1-983a-003fc9290167 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:45.212917 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-670da3a3-6bb0-48c1-983a-003fc9290167 None admin] RBAC: Adding query filter params (role.id=a6e562cdc96c4be691d638990f8403fd, scope.project.id=381cc34791e147329798cd454f99f622, user.id=97a0b159abf44cd59fe5a2b7bde28adf) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:19:45.213042 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-670da3a3-6bb0-48c1-983a-003fc9290167 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:45.214917 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-670da3a3-6bb0-48c1-983a-003fc9290167 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:45.228105 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 47/93] 10.222.0.90 () {56 vars in 1374 bytes} [Mon Aug 29 21:19:44 2022] GET /identity/v3/role_assignments?user.id=97a0b159abf44cd59fe5a2b7bde28adf&scope.project.id=381cc34791e147329798cd454f99f622&role.id=a6e562cdc96c4be691d638990f8403fd => generated 259 bytes in 278 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:19:45.228381 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2aa317c-4b06-4d22-8850-56fda664378d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:45.228931 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-670da3a3-6bb0-48c1-983a-003fc9290167 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:45.233810 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2aa317c-4b06-4d22-8850-56fda664378d None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:45.234483 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2aa317c-4b06-4d22-8850-56fda664378d None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:45.234964 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2aa317c-4b06-4d22-8850-56fda664378d None admin] PATH_INFO: `/v3/projects/381cc34791e147329798cd454f99f622` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:45.236547 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2aa317c-4b06-4d22-8850-56fda664378d None admin] RBAC: Authorizing `identity:get_project(project_id=381cc34791e147329798cd454f99f622)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:45.238373 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2aa317c-4b06-4d22-8850-56fda664378d None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:45.240863 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 47/94] 10.222.0.90 () {56 vars in 1186 bytes} [Mon Aug 29 21:19:45 2022] GET /identity/v3/projects/381cc34791e147329798cd454f99f622 => generated 298 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:19:45.251783 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2aa317c-4b06-4d22-8850-56fda664378d None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:45.425392 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2aa317c-4b06-4d22-8850-56fda664378d None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:45.463073 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:45.470102 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-398a7ad4-81ed-45a0-af45-bf14723d3ccd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:45.471033 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-398a7ad4-81ed-45a0-af45-bf14723d3ccd None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:45.471766 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-398a7ad4-81ed-45a0-af45-bf14723d3ccd None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:45.472239 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-398a7ad4-81ed-45a0-af45-bf14723d3ccd None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:45.472761 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-670da3a3-6bb0-48c1-983a-003fc9290167 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:45.473393 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-398a7ad4-81ed-45a0-af45-bf14723d3ccd None admin] RBAC: Adding query filter params (role.id=9745586777bf473bab7d887bf25bf236, scope.project.id=381cc34791e147329798cd454f99f622, user.id=94fd78dbbf6e44308bd6e72458d6361f) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:19:45.474672 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-398a7ad4-81ed-45a0-af45-bf14723d3ccd None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:45.477817 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-398a7ad4-81ed-45a0-af45-bf14723d3ccd None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:45.487798 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 48/95] 10.222.0.90 () {56 vars in 1374 bytes} [Mon Aug 29 21:19:45 2022] GET /identity/v3/role_assignments?user.id=94fd78dbbf6e44308bd6e72458d6361f&scope.project.id=381cc34791e147329798cd454f99f622&role.id=9745586777bf473bab7d887bf25bf236 => generated 259 bytes in 237 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:19:45.528346 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c920d74-cd9d-461f-82d9-7399450e501a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:45.529380 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c920d74-cd9d-461f-82d9-7399450e501a None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:45.529927 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c920d74-cd9d-461f-82d9-7399450e501a None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:45.530404 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c920d74-cd9d-461f-82d9-7399450e501a None admin] PATH_INFO: `/v3/projects/c7d7a59f097c47d89a7e42b80adb5b50` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:45.531806 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c920d74-cd9d-461f-82d9-7399450e501a None admin] RBAC: Authorizing `identity:get_project(project_id=c7d7a59f097c47d89a7e42b80adb5b50)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:45.533630 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c920d74-cd9d-461f-82d9-7399450e501a None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:45.535720 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 48/96] 10.222.0.90 () {56 vars in 1186 bytes} [Mon Aug 29 21:19:45 2022] GET /identity/v3/projects/c7d7a59f097c47d89a7e42b80adb5b50 => generated 312 bytes in 307 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:19:45.542518 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-398a7ad4-81ed-45a0-af45-bf14723d3ccd None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:45.545950 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:45.665403 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-398a7ad4-81ed-45a0-af45-bf14723d3ccd None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:45.703762 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a51c2c2-d039-416b-b69e-276f55e3d84b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:45.704863 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a51c2c2-d039-416b-b69e-276f55e3d84b None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:45.705369 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a51c2c2-d039-416b-b69e-276f55e3d84b None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:45.705871 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a51c2c2-d039-416b-b69e-276f55e3d84b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:45.706834 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a51c2c2-d039-416b-b69e-276f55e3d84b None admin] RBAC: Adding query filter params (role.id=02e242d5520740d4b77a5f4008c7eb57, scope.project.id=c7d7a59f097c47d89a7e42b80adb5b50, user.id=94fd78dbbf6e44308bd6e72458d6361f) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:19:45.707352 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a51c2c2-d039-416b-b69e-276f55e3d84b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:45.710398 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a51c2c2-d039-416b-b69e-276f55e3d84b None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:45.721025 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 49/97] 10.222.0.90 () {56 vars in 1374 bytes} [Mon Aug 29 21:19:45 2022] GET /identity/v3/role_assignments?user.id=94fd78dbbf6e44308bd6e72458d6361f&scope.project.id=c7d7a59f097c47d89a7e42b80adb5b50&role.id=02e242d5520740d4b77a5f4008c7eb57 => generated 259 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:19:46.735769 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4954a64d-d02c-419e-81c5-7c07b6198131 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:46.736624 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4954a64d-d02c-419e-81c5-7c07b6198131 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:46.737050 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4954a64d-d02c-419e-81c5-7c07b6198131 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:46.738617 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 49/98] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:19:46 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:19:46.744766 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eba15650-bbf9-46c0-b56f-e14d911a23c5 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:46.745506 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eba15650-bbf9-46c0-b56f-e14d911a23c5 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:46.745953 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eba15650-bbf9-46c0-b56f-e14d911a23c5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:46.760764 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f6f3196-40b4-4201-936c-3c7d65f60794 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:46.761088 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f6f3196-40b4-4201-936c-3c7d65f60794 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:46.761316 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f6f3196-40b4-4201-936c-3c7d65f60794 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:46.762807 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 50/99] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:19:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 29 21:19:46.768275 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d22d8a90-82bc-4bcd-b3cd-de6bf7542d07 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:46.768560 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d22d8a90-82bc-4bcd-b3cd-de6bf7542d07 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:46.768838 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d22d8a90-82bc-4bcd-b3cd-de6bf7542d07 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:46.786522 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-eba15650-bbf9-46c0-b56f-e14d911a23c5 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:19:46.787803 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-eba15650-bbf9-46c0-b56f-e14d911a23c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:19:46.810536 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-d22d8a90-82bc-4bcd-b3cd-de6bf7542d07 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:19:46.811735 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-d22d8a90-82bc-4bcd-b3cd-de6bf7542d07 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:19:46.930194 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:46.942386 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 50/100] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:19:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 199 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 21:19:46.953845 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 51/101] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:19:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 29 21:19:47.268160 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eba15650-bbf9-46c0-b56f-e14d911a23c5 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:47.400479 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eba15650-bbf9-46c0-b56f-e14d911a23c5 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:47.437057 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afc13685-6219-4ac8-a9d5-536b1fc7fbaa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:47.437703 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afc13685-6219-4ac8-a9d5-536b1fc7fbaa None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:47.438011 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afc13685-6219-4ac8-a9d5-536b1fc7fbaa None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:47.438287 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afc13685-6219-4ac8-a9d5-536b1fc7fbaa None admin] PATH_INFO: `/v3/domains/default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:47.439403 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afc13685-6219-4ac8-a9d5-536b1fc7fbaa None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:47.440862 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afc13685-6219-4ac8-a9d5-536b1fc7fbaa None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:47.443353 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 51/102] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:19:47 2022] GET /identity/v3/domains/default => generated 199 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:19:47.448760 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d22d8a90-82bc-4bcd-b3cd-de6bf7542d07 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:47.546378 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afc13685-6219-4ac8-a9d5-536b1fc7fbaa None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:47.588474 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:47.600565 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d22d8a90-82bc-4bcd-b3cd-de6bf7542d07 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:47.658509 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a91e33f-7883-4218-b633-b1e12d651543 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:47.659148 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a91e33f-7883-4218-b633-b1e12d651543 None admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:47.659442 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a91e33f-7883-4218-b633-b1e12d651543 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:47.659724 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a91e33f-7883-4218-b633-b1e12d651543 None admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:47.660414 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a91e33f-7883-4218-b633-b1e12d651543 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:47.662459 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a91e33f-7883-4218-b633-b1e12d651543 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:47.670208 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:47.711207 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 52/103] 10.222.0.90 () {60 vars in 1141 bytes} [Mon Aug 29 21:19:47 2022] POST /identity/v3/projects => generated 302 bytes in 262 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:19:47.733304 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afc13685-6219-4ac8-a9d5-536b1fc7fbaa None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:47.773701 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e346d3e0-8a49-4642-b99e-eac2f6bd4254 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:47.774394 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e346d3e0-8a49-4642-b99e-eac2f6bd4254 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:47.774651 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e346d3e0-8a49-4642-b99e-eac2f6bd4254 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:47.775232 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e346d3e0-8a49-4642-b99e-eac2f6bd4254 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:47.788212 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e346d3e0-8a49-4642-b99e-eac2f6bd4254 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 21:19:47.793600 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-e346d3e0-8a49-4642-b99e-eac2f6bd4254 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 29 21:19:47.793600 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:19:47.793600 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:19:47.793600 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:19:47.793600 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:19:47.793600 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:19:47.793600 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:19:47.793600 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:19:47.793600 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:19:47.793600 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:19:47.793600 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:19:47.793600 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:19:47.793600 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 21:19:47.793600 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 21:19:47.793600 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 21:19:47.793600 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 21:19:47.794554 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 21:19:47.794554 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 21:19:47.794554 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 21:19:47.794554 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 21:19:47.794554 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:19:47.794554 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:19:47.794554 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:19:47.794554 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:19:47.794554 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:19:47.794554 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:19:47.794554 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:19:47.794554 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:19:47.794554 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:19:47.794554 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:19:47.794554 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:19:47.794554 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:19:47.795334 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:19:47.795334 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:19:47.795334 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:19:47.795334 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:19:47.795334 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 21:19:47.795334 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 21:19:47.795334 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 21:19:47.795334 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 21:19:47.795334 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 21:19:47.795334 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 21:19:47.795334 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 29 21:19:47.795334 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:19:47.795334 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 52/104] 10.222.0.90 () {56 vars in 1099 bytes} [Mon Aug 29 21:19:47 2022] GET /identity/v3/roles/reader => generated 84 bytes in 249 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 29 21:19:47.804271 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a91e33f-7883-4218-b633-b1e12d651543 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:47.918869 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa9e7f3b-f2c6-4e28-aeaa-21c00551bd0e None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:47.919191 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa9e7f3b-f2c6-4e28-aeaa-21c00551bd0e None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:47.919424 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa9e7f3b-f2c6-4e28-aeaa-21c00551bd0e None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:47.921246 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 53/105] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:19:47 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:19:47.936381 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-990a4752-ce6f-4f1c-98ce-213ba540f9a0 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:47.936786 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-990a4752-ce6f-4f1c-98ce-213ba540f9a0 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:47.937070 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-990a4752-ce6f-4f1c-98ce-213ba540f9a0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:47.971067 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-990a4752-ce6f-4f1c-98ce-213ba540f9a0 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:19:47.972626 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-990a4752-ce6f-4f1c-98ce-213ba540f9a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:19:47.980433 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a91e33f-7883-4218-b633-b1e12d651543 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:48.029082 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a5f8196-0eb0-4db4-a9c2-02cd235fbb9f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:48.030077 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a5f8196-0eb0-4db4-a9c2-02cd235fbb9f None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:48.030622 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a5f8196-0eb0-4db4-a9c2-02cd235fbb9f None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:48.031101 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a5f8196-0eb0-4db4-a9c2-02cd235fbb9f None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:48.032054 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a5f8196-0eb0-4db4-a9c2-02cd235fbb9f None admin] RBAC: Adding query filter params (name=reader) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:19:48.032608 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a5f8196-0eb0-4db4-a9c2-02cd235fbb9f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:48.034221 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a5f8196-0eb0-4db4-a9c2-02cd235fbb9f None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:48.044355 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 53/106] 10.222.0.90 () {56 vars in 1101 bytes} [Mon Aug 29 21:19:47 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 241 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:19:48.046003 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a9dc44-dd76-484a-b56f-df7cc4c8dc56 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:48.046484 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a9dc44-dd76-484a-b56f-df7cc4c8dc56 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:48.046971 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a9dc44-dd76-484a-b56f-df7cc4c8dc56 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:48.049393 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 54/107] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:19:48 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:19:48.050354 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53a9dc44-dd76-484a-b56f-df7cc4c8dc56 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:48.070938 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 54/108] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:19:47 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 29 21:19:48.077826 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3742119-3444-4ab8-a035-a7eb12c0cd43 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:48.077826 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3742119-3444-4ab8-a035-a7eb12c0cd43 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:48.077826 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3742119-3444-4ab8-a035-a7eb12c0cd43 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:48.078164 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:48.106595 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-a3742119-3444-4ab8-a035-a7eb12c0cd43 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:19:48.108016 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-a3742119-3444-4ab8-a035-a7eb12c0cd43 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:19:48.169409 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:48.190955 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 55/109] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:19:48 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 21:19:48.254053 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53a9dc44-dd76-484a-b56f-df7cc4c8dc56 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:48.299861 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8acd1d2b-c8ae-4ae1-a693-beaff5aa45f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:48.300876 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8acd1d2b-c8ae-4ae1-a693-beaff5aa45f7 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:48.301356 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8acd1d2b-c8ae-4ae1-a693-beaff5aa45f7 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:48.302229 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8acd1d2b-c8ae-4ae1-a693-beaff5aa45f7 None admin] PATH_INFO: `/v3/users/b85a8c7cd85e4ae38ddbc5f2b5af87c5` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:48.322754 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8acd1d2b-c8ae-4ae1-a693-beaff5aa45f7 None admin] RBAC: Authorizing `identity:get_user(user_id=b85a8c7cd85e4ae38ddbc5f2b5af87c5)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:48.324661 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8acd1d2b-c8ae-4ae1-a693-beaff5aa45f7 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:48.326680 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 55/110] 10.222.0.90 () {56 vars in 1177 bytes} [Mon Aug 29 21:19:48 2022] GET /identity/v3/users/b85a8c7cd85e4ae38ddbc5f2b5af87c5 => generated 289 bytes in 277 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:19:48.332452 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3742119-3444-4ab8-a035-a7eb12c0cd43 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:48.461534 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3742119-3444-4ab8-a035-a7eb12c0cd43 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:48.499319 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4d70031-2675-4395-a691-cf20af81b32b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:48.499956 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d70031-2675-4395-a691-cf20af81b32b None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:48.500247 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d70031-2675-4395-a691-cf20af81b32b None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:48.500530 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d70031-2675-4395-a691-cf20af81b32b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:48.537172 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a066fd17-96ee-4161-b1cd-1a759da7650d None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:48.537821 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a066fd17-96ee-4161-b1cd-1a759da7650d None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:48.538293 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a066fd17-96ee-4161-b1cd-1a759da7650d None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:48.539930 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 56/111] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:19:48 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:19:48.545257 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ace73c1-f541-4d9b-87dc-1537b3e2e917 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:48.545870 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ace73c1-f541-4d9b-87dc-1537b3e2e917 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:48.546358 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ace73c1-f541-4d9b-87dc-1537b3e2e917 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:48.561178 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:48.568710 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-9ace73c1-f541-4d9b-87dc-1537b3e2e917 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:19:48.570913 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-9ace73c1-f541-4d9b-87dc-1537b3e2e917 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:19:48.596605 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4d70031-2675-4395-a691-cf20af81b32b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:48.600165 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4d70031-2675-4395-a691-cf20af81b32b None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:48.696837 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:48.706840 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 57/112] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:19:48 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 29 21:19:48.708611 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ace73c1-f541-4d9b-87dc-1537b3e2e917 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:48.761752 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 56/113] 10.222.0.90 () {58 vars in 1282 bytes} [Mon Aug 29 21:19:48 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 427 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 29 21:19:48.761752 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4d70031-2675-4395-a691-cf20af81b32b None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:48.846587 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ace73c1-f541-4d9b-87dc-1537b3e2e917 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:48.852227 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:48.870631 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4d70031-2675-4395-a691-cf20af81b32b None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:48.888887 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-841a333f-e3ff-4728-a9f5-acb16d3fa16c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:48.889972 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-841a333f-e3ff-4728-a9f5-acb16d3fa16c None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:48.890450 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-841a333f-e3ff-4728-a9f5-acb16d3fa16c None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:48.890936 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-841a333f-e3ff-4728-a9f5-acb16d3fa16c None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:48.903420 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-841a333f-e3ff-4728-a9f5-acb16d3fa16c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 21:19:48.908771 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d196b731-3717-4c57-a645-eb7a35d29c93 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:48.908927 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-841a333f-e3ff-4728-a9f5-acb16d3fa16c None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 29 21:19:48.908927 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:19:48.908927 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:19:48.908927 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:19:48.908927 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:19:48.908927 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:19:48.908927 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:19:48.908927 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:19:48.908927 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:19:48.908927 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:19:48.908927 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:19:48.908927 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:19:48.908927 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 21:19:48.908927 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 21:19:48.908927 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 21:19:48.908927 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 21:19:48.909727 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 21:19:48.909727 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 21:19:48.909727 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 21:19:48.909727 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 21:19:48.909727 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:19:48.909727 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:19:48.909727 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:19:48.909727 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:19:48.909727 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:19:48.909727 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:19:48.909727 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:19:48.909727 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:19:48.909727 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:19:48.909727 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:19:48.909727 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:19:48.909727 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:19:48.910533 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:19:48.910533 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:19:48.910533 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:19:48.910533 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:19:48.910533 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 21:19:48.910533 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 21:19:48.910533 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 21:19:48.910533 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 21:19:48.910533 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 21:19:48.910533 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 21:19:48.910533 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 29 21:19:48.910533 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:19:48.911852 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d196b731-3717-4c57-a645-eb7a35d29c93 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:48.911852 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d196b731-3717-4c57-a645-eb7a35d29c93 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:48.911852 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d196b731-3717-4c57-a645-eb7a35d29c93 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:48.914172 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 58/114] 10.222.0.90 () {56 vars in 1114 bytes} [Mon Aug 29 21:19:48 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:19:48.915243 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-841a333f-e3ff-4728-a9f5-acb16d3fa16c None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:48.934771 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d196b731-3717-4c57-a645-eb7a35d29c93 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 21:19:48.944402 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-d196b731-3717-4c57-a645-eb7a35d29c93 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 29 21:19:48.944402 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:19:48.944402 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:19:48.944402 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:19:48.944402 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:19:48.944402 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:19:48.944402 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:19:48.944402 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:19:48.944402 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:19:48.944402 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:19:48.944402 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:19:48.944402 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:19:48.944402 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 21:19:48.944402 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 21:19:48.944402 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 21:19:48.944402 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 21:19:48.945294 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 21:19:48.945294 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 21:19:48.945294 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 21:19:48.945294 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 21:19:48.945294 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:19:48.945294 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:19:48.945294 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:19:48.945294 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:19:48.945294 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:19:48.945294 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:19:48.945294 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:19:48.945294 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:19:48.945294 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:19:48.945294 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:19:48.945294 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:19:48.945294 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:19:48.946125 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:19:48.946125 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:19:48.946125 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:19:48.946125 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:19:48.946125 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 21:19:48.946125 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 21:19:48.946125 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 21:19:48.946125 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 21:19:48.946125 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 21:19:48.946125 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 21:19:48.946125 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 29 21:19:48.946125 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:19:48.946125 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 57/115] 10.222.0.90 () {56 vars in 1099 bytes} [Mon Aug 29 21:19:48 2022] GET /identity/v3/roles/member => generated 84 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:19:48.946775 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d196b731-3717-4c57-a645-eb7a35d29c93 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:48.986311 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:49.032608 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-841a333f-e3ff-4728-a9f5-acb16d3fa16c None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:49.070512 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b79189c6-5a4f-45ce-9b41-4f560fbbb8e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:49.071156 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b79189c6-5a4f-45ce-9b41-4f560fbbb8e1 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:49.071438 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b79189c6-5a4f-45ce-9b41-4f560fbbb8e1 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:49.071738 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b79189c6-5a4f-45ce-9b41-4f560fbbb8e1 None admin] PATH_INFO: `/v3/projects/381cc34791e147329798cd454f99f622` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:49.072870 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b79189c6-5a4f-45ce-9b41-4f560fbbb8e1 None admin] RBAC: Authorizing `identity:get_project(project_id=381cc34791e147329798cd454f99f622)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:49.074366 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b79189c6-5a4f-45ce-9b41-4f560fbbb8e1 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:49.075997 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 59/116] 10.222.0.90 () {56 vars in 1186 bytes} [Mon Aug 29 21:19:48 2022] GET /identity/v3/projects/381cc34791e147329798cd454f99f622 => generated 298 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:19:49.076754 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b79189c6-5a4f-45ce-9b41-4f560fbbb8e1 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:49.126634 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d196b731-3717-4c57-a645-eb7a35d29c93 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:49.163473 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66660995-fab7-492d-b4e5-9a18f92c8b87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:49.164466 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66660995-fab7-492d-b4e5-9a18f92c8b87 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:49.164972 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66660995-fab7-492d-b4e5-9a18f92c8b87 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:49.165444 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66660995-fab7-492d-b4e5-9a18f92c8b87 None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:49.166371 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66660995-fab7-492d-b4e5-9a18f92c8b87 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:19:49.166876 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66660995-fab7-492d-b4e5-9a18f92c8b87 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:49.168449 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66660995-fab7-492d-b4e5-9a18f92c8b87 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:49.177517 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 58/117] 10.222.0.90 () {56 vars in 1111 bytes} [Mon Aug 29 21:19:48 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 231 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:19:49.179227 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e78fde3-a154-46a3-9223-5e844142a3d6 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:49.179716 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e78fde3-a154-46a3-9223-5e844142a3d6 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:49.179949 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b79189c6-5a4f-45ce-9b41-4f560fbbb8e1 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:49.180399 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e78fde3-a154-46a3-9223-5e844142a3d6 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:49.182028 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 59/118] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:19:49 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:19:49.186892 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e78fde3-a154-46a3-9223-5e844142a3d6 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:49.196859 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:49.205347 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:49.218027 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50f6d59a-4bb8-4768-a763-ece6aa518107 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:49.218640 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50f6d59a-4bb8-4768-a763-ece6aa518107 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:49.218922 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50f6d59a-4bb8-4768-a763-ece6aa518107 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:49.219400 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50f6d59a-4bb8-4768-a763-ece6aa518107 None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:49.219863 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50f6d59a-4bb8-4768-a763-ece6aa518107 None admin] RBAC: Adding query filter params (name=member) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:19:49.220167 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50f6d59a-4bb8-4768-a763-ece6aa518107 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:49.221891 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50f6d59a-4bb8-4768-a763-ece6aa518107 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:49.235075 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 60/119] 10.222.0.90 () {56 vars in 1101 bytes} [Mon Aug 29 21:19:49 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:19:49.235075 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50f6d59a-4bb8-4768-a763-ece6aa518107 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:49.308642 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e78fde3-a154-46a3-9223-5e844142a3d6 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:49.342379 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50f6d59a-4bb8-4768-a763-ece6aa518107 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:49.347041 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f12e1cdd-c04a-4378-9430-ce29384bf75d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:49.347663 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f12e1cdd-c04a-4378-9430-ce29384bf75d None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:49.347947 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f12e1cdd-c04a-4378-9430-ce29384bf75d None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:49.348223 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f12e1cdd-c04a-4378-9430-ce29384bf75d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:49.348993 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f12e1cdd-c04a-4378-9430-ce29384bf75d None admin] RBAC: Adding query filter params (role.id=414a14ad48a345f89fd1045f5d8fd16c, scope.project.id=381cc34791e147329798cd454f99f622, user.id=b85a8c7cd85e4ae38ddbc5f2b5af87c5) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:19:49.349306 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f12e1cdd-c04a-4378-9430-ce29384bf75d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:49.350836 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f12e1cdd-c04a-4378-9430-ce29384bf75d None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:49.356456 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 60/120] 10.222.0.90 () {56 vars in 1374 bytes} [Mon Aug 29 21:19:49 2022] GET /identity/v3/role_assignments?user.id=b85a8c7cd85e4ae38ddbc5f2b5af87c5&scope.project.id=381cc34791e147329798cd454f99f622&role.id=414a14ad48a345f89fd1045f5d8fd16c => generated 259 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:19:49.357246 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f12e1cdd-c04a-4378-9430-ce29384bf75d None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:49.394681 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53202c49-18b3-43ac-9793-9a249afd261b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:49.396338 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53202c49-18b3-43ac-9793-9a249afd261b None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:49.396842 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53202c49-18b3-43ac-9793-9a249afd261b None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:49.397312 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53202c49-18b3-43ac-9793-9a249afd261b None admin] PATH_INFO: `/v3/users/94fd78dbbf6e44308bd6e72458d6361f` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:49.398782 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53202c49-18b3-43ac-9793-9a249afd261b None admin] RBAC: Authorizing `identity:get_user(user_id=94fd78dbbf6e44308bd6e72458d6361f)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:49.400483 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53202c49-18b3-43ac-9793-9a249afd261b None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:49.402553 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 61/121] 10.222.0.90 () {56 vars in 1177 bytes} [Mon Aug 29 21:19:49 2022] GET /identity/v3/users/94fd78dbbf6e44308bd6e72458d6361f => generated 275 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:19:49.404066 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d4d679-783c-4508-bbca-df65f0f9d8d3 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:49.404564 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d4d679-783c-4508-bbca-df65f0f9d8d3 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:49.405124 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d4d679-783c-4508-bbca-df65f0f9d8d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:49.427496 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-28d4d679-783c-4508-bbca-df65f0f9d8d3 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:19:49.429215 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-28d4d679-783c-4508-bbca-df65f0f9d8d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:19:49.442702 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:49.481496 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f12e1cdd-c04a-4378-9430-ce29384bf75d None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:49.489334 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:49.510270 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 62/122] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:19:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 21:19:49.511440 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28d4d679-783c-4508-bbca-df65f0f9d8d3 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:49.521264 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b24af3b-05f3-4ce5-8a1d-95e785d17b63 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:49.521463 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b24af3b-05f3-4ce5-8a1d-95e785d17b63 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:49.522515 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b24af3b-05f3-4ce5-8a1d-95e785d17b63 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:49.522864 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b24af3b-05f3-4ce5-8a1d-95e785d17b63 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:49.534823 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b24af3b-05f3-4ce5-8a1d-95e785d17b63 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 21:19:49.541121 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-0b24af3b-05f3-4ce5-8a1d-95e785d17b63 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 21:19:49.541121 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:19:49.541121 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:19:49.541121 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:19:49.541121 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:19:49.541121 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:19:49.541121 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:19:49.541121 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:19:49.541121 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:19:49.541121 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:19:49.541121 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:19:49.541121 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:19:49.541121 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 21:19:49.541121 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 21:19:49.541121 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 21:19:49.541121 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 21:19:49.542028 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 21:19:49.542028 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 21:19:49.542028 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 21:19:49.542028 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 21:19:49.542028 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:19:49.542028 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:19:49.542028 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:19:49.542028 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:19:49.542028 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:19:49.542028 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:19:49.542028 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:19:49.542028 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:19:49.542028 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:19:49.542028 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:19:49.542028 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:19:49.542028 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:19:49.542905 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:19:49.542905 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:19:49.542905 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:19:49.542905 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:19:49.542905 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 21:19:49.542905 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 21:19:49.542905 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 21:19:49.542905 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 21:19:49.542905 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 21:19:49.542905 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 21:19:49.542905 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 21:19:49.542905 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:19:49.542905 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 61/123] 10.222.0.90 () {56 vars in 1096 bytes} [Mon Aug 29 21:19:49 2022] GET /identity/v3/roles/admin => generated 83 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:19:49.543584 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b24af3b-05f3-4ce5-8a1d-95e785d17b63 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:49.622901 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28d4d679-783c-4508-bbca-df65f0f9d8d3 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:49.656442 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b24af3b-05f3-4ce5-8a1d-95e785d17b63 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:49.659950 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adce7a07-86cc-435c-a556-fa206993d3b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:49.660752 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adce7a07-86cc-435c-a556-fa206993d3b3 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:49.661369 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adce7a07-86cc-435c-a556-fa206993d3b3 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:49.661880 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adce7a07-86cc-435c-a556-fa206993d3b3 None admin] PATH_INFO: `/v3/users/94fd78dbbf6e44308bd6e72458d6361f` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:49.663243 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adce7a07-86cc-435c-a556-fa206993d3b3 None admin] RBAC: Authorizing `identity:get_user(user_id=94fd78dbbf6e44308bd6e72458d6361f)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:49.664863 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adce7a07-86cc-435c-a556-fa206993d3b3 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:49.666698 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 63/124] 10.222.0.90 () {56 vars in 1177 bytes} [Mon Aug 29 21:19:49 2022] GET /identity/v3/users/94fd78dbbf6e44308bd6e72458d6361f => generated 275 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:19:49.667582 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adce7a07-86cc-435c-a556-fa206993d3b3 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:49.688107 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:49.696925 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47db505d-8047-463d-941a-707e75481804 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:49.697970 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47db505d-8047-463d-941a-707e75481804 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:49.698518 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47db505d-8047-463d-941a-707e75481804 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:49.698993 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47db505d-8047-463d-941a-707e75481804 None admin] PATH_INFO: `/v3/projects/381cc34791e147329798cd454f99f622` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:49.700369 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47db505d-8047-463d-941a-707e75481804 None admin] RBAC: Authorizing `identity:get_project(project_id=381cc34791e147329798cd454f99f622)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:49.702072 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47db505d-8047-463d-941a-707e75481804 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:49.704115 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 62/125] 10.222.0.90 () {56 vars in 1186 bytes} [Mon Aug 29 21:19:49 2022] GET /identity/v3/projects/381cc34791e147329798cd454f99f622 => generated 298 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:19:49.706017 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47db505d-8047-463d-941a-707e75481804 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:49.728397 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:49.779920 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adce7a07-86cc-435c-a556-fa206993d3b3 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:49.816480 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47db505d-8047-463d-941a-707e75481804 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:49.817361 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5ea0ce0-d986-437b-80d6-9a542b0d0946 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:49.817887 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5ea0ce0-d986-437b-80d6-9a542b0d0946 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:49.818239 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5ea0ce0-d986-437b-80d6-9a542b0d0946 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:49.818527 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5ea0ce0-d986-437b-80d6-9a542b0d0946 None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:49.819201 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5ea0ce0-d986-437b-80d6-9a542b0d0946 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:19:49.819504 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5ea0ce0-d986-437b-80d6-9a542b0d0946 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:49.820884 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5ea0ce0-d986-437b-80d6-9a542b0d0946 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:49.830287 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 64/126] 10.222.0.90 () {56 vars in 1099 bytes} [Mon Aug 29 21:19:49 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:19:49.831129 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5ea0ce0-d986-437b-80d6-9a542b0d0946 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:49.860840 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d897b8fb-c776-4fbc-be01-75a51aa3139a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:49.862231 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d897b8fb-c776-4fbc-be01-75a51aa3139a None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:49.862736 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d897b8fb-c776-4fbc-be01-75a51aa3139a None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:49.863211 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d897b8fb-c776-4fbc-be01-75a51aa3139a None admin] PATH_INFO: `/v3/projects/381cc34791e147329798cd454f99f622` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:49.865198 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d897b8fb-c776-4fbc-be01-75a51aa3139a None admin] RBAC: Authorizing `identity:get_project(project_id=381cc34791e147329798cd454f99f622)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:49.867597 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d897b8fb-c776-4fbc-be01-75a51aa3139a None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:49.869481 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 63/127] 10.222.0.90 () {56 vars in 1186 bytes} [Mon Aug 29 21:19:49 2022] GET /identity/v3/projects/381cc34791e147329798cd454f99f622 => generated 298 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:19:49.870509 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d897b8fb-c776-4fbc-be01-75a51aa3139a None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:49.978430 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5ea0ce0-d986-437b-80d6-9a542b0d0946 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:49.979206 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:50.004320 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:50.021151 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d897b8fb-c776-4fbc-be01-75a51aa3139a None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:50.022659 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cad7f4db-8fb1-444b-b2b4-391d9cb3c423 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:50.023606 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cad7f4db-8fb1-444b-b2b4-391d9cb3c423 None admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:50.024087 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cad7f4db-8fb1-444b-b2b4-391d9cb3c423 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:50.024556 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cad7f4db-8fb1-444b-b2b4-391d9cb3c423 None admin] PATH_INFO: `/v3/projects/381cc34791e147329798cd454f99f622/users/94fd78dbbf6e44308bd6e72458d6361f/roles/9745586777bf473bab7d887bf25bf236` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:50.027014 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cad7f4db-8fb1-444b-b2b4-391d9cb3c423 None admin] RBAC: Authorizing `identity:create_grant(project_id=381cc34791e147329798cd454f99f622, user_id=94fd78dbbf6e44308bd6e72458d6361f, role_id=9745586777bf473bab7d887bf25bf236)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:50.028727 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cad7f4db-8fb1-444b-b2b4-391d9cb3c423 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:50.045025 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 65/128] 10.222.0.90 () {58 vars in 1439 bytes} [Mon Aug 29 21:19:49 2022] PUT /identity/v3/projects/381cc34791e147329798cd454f99f622/users/94fd78dbbf6e44308bd6e72458d6361f/roles/9745586777bf473bab7d887bf25bf236 => generated 0 bytes in 214 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:19:50.046885 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cad7f4db-8fb1-444b-b2b4-391d9cb3c423 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:50.065477 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b93dcff8-64d4-400a-80d2-c564fb7004c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:50.066592 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b93dcff8-64d4-400a-80d2-c564fb7004c2 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:50.066967 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b93dcff8-64d4-400a-80d2-c564fb7004c2 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:50.067808 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b93dcff8-64d4-400a-80d2-c564fb7004c2 None admin] PATH_INFO: `/v3/users/97a0b159abf44cd59fe5a2b7bde28adf` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:50.069133 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b93dcff8-64d4-400a-80d2-c564fb7004c2 None admin] RBAC: Authorizing `identity:get_user(user_id=97a0b159abf44cd59fe5a2b7bde28adf)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:50.070964 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b93dcff8-64d4-400a-80d2-c564fb7004c2 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:50.073664 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 64/129] 10.222.0.90 () {56 vars in 1177 bytes} [Mon Aug 29 21:19:49 2022] GET /identity/v3/users/97a0b159abf44cd59fe5a2b7bde28adf => generated 247 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:19:50.078265 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b93dcff8-64d4-400a-80d2-c564fb7004c2 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:50.200944 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cad7f4db-8fb1-444b-b2b4-391d9cb3c423 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:50.242977 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1063200b-d77c-42ef-a815-195dcb0e5da2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:50.243691 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1063200b-d77c-42ef-a815-195dcb0e5da2 None admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:50.243984 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1063200b-d77c-42ef-a815-195dcb0e5da2 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:50.244267 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1063200b-d77c-42ef-a815-195dcb0e5da2 None admin] PATH_INFO: `/v3/projects/381cc34791e147329798cd454f99f622/users/94fd78dbbf6e44308bd6e72458d6361f/roles/02e242d5520740d4b77a5f4008c7eb57` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:50.246419 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1063200b-d77c-42ef-a815-195dcb0e5da2 None admin] RBAC: Authorizing `identity:create_grant(project_id=381cc34791e147329798cd454f99f622, user_id=94fd78dbbf6e44308bd6e72458d6361f, role_id=02e242d5520740d4b77a5f4008c7eb57)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:50.248060 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1063200b-d77c-42ef-a815-195dcb0e5da2 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:50.266484 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 66/130] 10.222.0.90 () {58 vars in 1439 bytes} [Mon Aug 29 21:19:50 2022] PUT /identity/v3/projects/381cc34791e147329798cd454f99f622/users/94fd78dbbf6e44308bd6e72458d6361f/roles/02e242d5520740d4b77a5f4008c7eb57 => generated 0 bytes in 220 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:19:50.267488 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1063200b-d77c-42ef-a815-195dcb0e5da2 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:50.267760 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b93dcff8-64d4-400a-80d2-c564fb7004c2 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:50.313004 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:50.330680 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80886423-dbc8-40fe-9fc8-ca24a7e3b512 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:50.331362 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80886423-dbc8-40fe-9fc8-ca24a7e3b512 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:50.331647 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80886423-dbc8-40fe-9fc8-ca24a7e3b512 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:50.331926 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80886423-dbc8-40fe-9fc8-ca24a7e3b512 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:50.335159 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:50.351811 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80886423-dbc8-40fe-9fc8-ca24a7e3b512 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 21:19:50.367685 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-80886423-dbc8-40fe-9fc8-ca24a7e3b512 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 29 21:19:50.367685 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:19:50.367685 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:19:50.367685 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:19:50.367685 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:19:50.367685 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:19:50.367685 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:19:50.367685 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:19:50.367685 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:19:50.367685 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:19:50.367685 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:19:50.367685 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:19:50.367685 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 21:19:50.367685 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 21:19:50.367685 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 21:19:50.367685 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 21:19:50.368669 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 21:19:50.368669 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 21:19:50.368669 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 21:19:50.368669 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 21:19:50.368669 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:19:50.368669 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:19:50.368669 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:19:50.368669 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:19:50.368669 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:19:50.368669 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:19:50.368669 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:19:50.368669 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:19:50.368669 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:19:50.368669 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:19:50.368669 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:19:50.368669 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:19:50.369492 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:19:50.369492 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:19:50.369492 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:19:50.369492 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:19:50.369492 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 21:19:50.369492 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 21:19:50.369492 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 21:19:50.369492 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 21:19:50.369492 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 21:19:50.369492 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 21:19:50.369492 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 29 21:19:50.369492 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:19:50.369492 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 65/131] 10.222.0.90 () {56 vars in 1099 bytes} [Mon Aug 29 21:19:50 2022] GET /identity/v3/roles/member => generated 84 bytes in 295 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:19:50.386905 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80886423-dbc8-40fe-9fc8-ca24a7e3b512 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:50.413045 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1063200b-d77c-42ef-a815-195dcb0e5da2 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:50.455160 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc4915c5-2592-4fe8-9754-36bb8d49c4c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:50.455815 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc4915c5-2592-4fe8-9754-36bb8d49c4c1 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:50.456101 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc4915c5-2592-4fe8-9754-36bb8d49c4c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:50.456381 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc4915c5-2592-4fe8-9754-36bb8d49c4c1 None admin] PATH_INFO: `/v3/projects/381cc34791e147329798cd454f99f622` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:50.457694 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc4915c5-2592-4fe8-9754-36bb8d49c4c1 None admin] RBAC: Authorizing `identity:get_project(project_id=381cc34791e147329798cd454f99f622)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:50.460372 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc4915c5-2592-4fe8-9754-36bb8d49c4c1 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:50.462286 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 67/132] 10.222.0.90 () {56 vars in 1186 bytes} [Mon Aug 29 21:19:50 2022] GET /identity/v3/projects/381cc34791e147329798cd454f99f622 => generated 298 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:19:50.474079 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc4915c5-2592-4fe8-9754-36bb8d49c4c1 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:50.608764 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80886423-dbc8-40fe-9fc8-ca24a7e3b512 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:50.661489 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc4915c5-2592-4fe8-9754-36bb8d49c4c1 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:50.663535 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:50.686195 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41b62556-d305-4523-8ebc-b1b03fcf9b7f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:50.686886 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b62556-d305-4523-8ebc-b1b03fcf9b7f None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:50.687179 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b62556-d305-4523-8ebc-b1b03fcf9b7f None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:50.687537 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b62556-d305-4523-8ebc-b1b03fcf9b7f None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:50.688142 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41b62556-d305-4523-8ebc-b1b03fcf9b7f None admin] RBAC: Adding query filter params (name=member) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:19:50.688452 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41b62556-d305-4523-8ebc-b1b03fcf9b7f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:50.689901 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41b62556-d305-4523-8ebc-b1b03fcf9b7f None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:50.705084 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 66/133] 10.222.0.90 () {56 vars in 1101 bytes} [Mon Aug 29 21:19:50 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 317 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:19:50.706039 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e56ccc86-9de7-4348-af06-803d7c135245 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:50.708648 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e56ccc86-9de7-4348-af06-803d7c135245 None admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:50.709162 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc836e53-078d-4e0c-9bcd-e73aab7207f0 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:50.710161 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc836e53-078d-4e0c-9bcd-e73aab7207f0 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:50.711604 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e56ccc86-9de7-4348-af06-803d7c135245 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:50.712150 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc836e53-078d-4e0c-9bcd-e73aab7207f0 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:50.713869 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 67/134] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:19:50 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:19:50.714443 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e56ccc86-9de7-4348-af06-803d7c135245 None admin] PATH_INFO: `/v3/projects/381cc34791e147329798cd454f99f622/users/97a0b159abf44cd59fe5a2b7bde28adf/roles/a6e562cdc96c4be691d638990f8403fd` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:50.716365 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc836e53-078d-4e0c-9bcd-e73aab7207f0 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:50.718198 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e56ccc86-9de7-4348-af06-803d7c135245 None admin] RBAC: Authorizing `identity:create_grant(project_id=381cc34791e147329798cd454f99f622, user_id=97a0b159abf44cd59fe5a2b7bde28adf, role_id=a6e562cdc96c4be691d638990f8403fd)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:50.719921 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e56ccc86-9de7-4348-af06-803d7c135245 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:50.748442 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 68/135] 10.222.0.90 () {58 vars in 1439 bytes} [Mon Aug 29 21:19:50 2022] PUT /identity/v3/projects/381cc34791e147329798cd454f99f622/users/97a0b159abf44cd59fe5a2b7bde28adf/roles/a6e562cdc96c4be691d638990f8403fd => generated 0 bytes in 280 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 21:19:50.750073 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2cce600-67d6-4b8f-af2c-9b10869cb24d None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:50.750562 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2cce600-67d6-4b8f-af2c-9b10869cb24d None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:50.751523 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2cce600-67d6-4b8f-af2c-9b10869cb24d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:50.800772 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-d2cce600-67d6-4b8f-af2c-9b10869cb24d None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:19:50.802810 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-d2cce600-67d6-4b8f-af2c-9b10869cb24d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:19:50.859655 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:50.943985 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc836e53-078d-4e0c-9bcd-e73aab7207f0 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:50.961226 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 69/136] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:19:50 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 212 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 21:19:51.013350 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e44406b-bead-49a3-a223-f4da677272af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:51.014042 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e44406b-bead-49a3-a223-f4da677272af None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:51.014331 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e44406b-bead-49a3-a223-f4da677272af None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:51.014611 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e44406b-bead-49a3-a223-f4da677272af None admin] PATH_INFO: `/v3/users/94fd78dbbf6e44308bd6e72458d6361f` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:51.015775 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e44406b-bead-49a3-a223-f4da677272af None admin] RBAC: Authorizing `identity:get_user(user_id=94fd78dbbf6e44308bd6e72458d6361f)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:51.017260 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e44406b-bead-49a3-a223-f4da677272af None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:51.018982 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 68/137] 10.222.0.90 () {56 vars in 1177 bytes} [Mon Aug 29 21:19:50 2022] GET /identity/v3/users/94fd78dbbf6e44308bd6e72458d6361f => generated 275 bytes in 303 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:19:51.026133 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2cce600-67d6-4b8f-af2c-9b10869cb24d None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:51.146649 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:51.155849 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2cce600-67d6-4b8f-af2c-9b10869cb24d None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:51.196089 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd3d2435-412f-47ca-9654-851bc5ebd47c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:51.196741 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd3d2435-412f-47ca-9654-851bc5ebd47c None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:51.197030 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd3d2435-412f-47ca-9654-851bc5ebd47c None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:51.197311 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd3d2435-412f-47ca-9654-851bc5ebd47c None admin] PATH_INFO: `/v3/projects/c7d7a59f097c47d89a7e42b80adb5b50` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:51.198547 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd3d2435-412f-47ca-9654-851bc5ebd47c None admin] RBAC: Authorizing `identity:get_project(project_id=c7d7a59f097c47d89a7e42b80adb5b50)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:51.200060 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd3d2435-412f-47ca-9654-851bc5ebd47c None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:51.202146 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 70/138] 10.222.0.90 () {56 vars in 1186 bytes} [Mon Aug 29 21:19:51 2022] GET /identity/v3/projects/c7d7a59f097c47d89a7e42b80adb5b50 => generated 312 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:19:51.211067 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e44406b-bead-49a3-a223-f4da677272af None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:51.346998 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e44406b-bead-49a3-a223-f4da677272af None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:51.357958 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd3d2435-412f-47ca-9654-851bc5ebd47c None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:51.383382 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:51.393293 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7a996a9-d018-415b-a547-562489315456 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:51.394011 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7a996a9-d018-415b-a547-562489315456 None admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:51.394298 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7a996a9-d018-415b-a547-562489315456 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:51.394577 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7a996a9-d018-415b-a547-562489315456 None admin] PATH_INFO: `/v3/projects/c7d7a59f097c47d89a7e42b80adb5b50/users/94fd78dbbf6e44308bd6e72458d6361f/roles/02e242d5520740d4b77a5f4008c7eb57` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:51.397173 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7a996a9-d018-415b-a547-562489315456 None admin] RBAC: Authorizing `identity:create_grant(project_id=c7d7a59f097c47d89a7e42b80adb5b50, user_id=94fd78dbbf6e44308bd6e72458d6361f, role_id=02e242d5520740d4b77a5f4008c7eb57)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:51.402672 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7a996a9-d018-415b-a547-562489315456 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:51.425606 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 69/139] 10.222.0.90 () {58 vars in 1439 bytes} [Mon Aug 29 21:19:51 2022] PUT /identity/v3/projects/c7d7a59f097c47d89a7e42b80adb5b50/users/94fd78dbbf6e44308bd6e72458d6361f/roles/02e242d5520740d4b77a5f4008c7eb57 => generated 0 bytes in 215 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:19:51.504206 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd3d2435-412f-47ca-9654-851bc5ebd47c None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:51.542966 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f8a3f13-aa88-4085-8009-caf0fb4de659 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:51.543626 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f8a3f13-aa88-4085-8009-caf0fb4de659 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:51.543907 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f8a3f13-aa88-4085-8009-caf0fb4de659 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:51.544181 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f8a3f13-aa88-4085-8009-caf0fb4de659 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:51.545348 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f8a3f13-aa88-4085-8009-caf0fb4de659 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:51.546920 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f8a3f13-aa88-4085-8009-caf0fb4de659 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:51.548906 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 71/140] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:19:51 2022] GET /identity/v3/domains/default => generated 199 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:19:51.555518 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7a996a9-d018-415b-a547-562489315456 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:51.698958 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7a996a9-d018-415b-a547-562489315456 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:51.747213 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbc273a2-e5cd-4a06-bbb5-8b4df4052aaf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:51.747691 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc273a2-e5cd-4a06-bbb5-8b4df4052aaf None admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:51.747977 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc273a2-e5cd-4a06-bbb5-8b4df4052aaf None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:51.748256 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc273a2-e5cd-4a06-bbb5-8b4df4052aaf None admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:51.748949 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbc273a2-e5cd-4a06-bbb5-8b4df4052aaf None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:51.750436 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbc273a2-e5cd-4a06-bbb5-8b4df4052aaf None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:51.781951 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 70/141] 10.222.0.90 () {60 vars in 1132 bytes} [Mon Aug 29 21:19:51 2022] POST /identity/v3/users => generated 283 bytes in 227 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:19:51.899307 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7487d288-1153-473b-b68a-493b9455ffd6 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:51.899639 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7487d288-1153-473b-b68a-493b9455ffd6 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:51.899865 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7487d288-1153-473b-b68a-493b9455ffd6 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:51.901564 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 72/142] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:19:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:19:51.907085 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7314b668-bbd2-4420-affb-c28396c69556 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:51.907371 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7314b668-bbd2-4420-affb-c28396c69556 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:51.907661 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7314b668-bbd2-4420-affb-c28396c69556 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:51.930013 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-7314b668-bbd2-4420-affb-c28396c69556 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:19:51.931244 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-7314b668-bbd2-4420-affb-c28396c69556 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:19:51.976600 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:52.020046 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 71/143] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:19:51 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 21:19:52.541230 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7487d288-1153-473b-b68a-493b9455ffd6 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:52.577638 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:52.684504 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7487d288-1153-473b-b68a-493b9455ffd6 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:52.725213 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-453e58d2-2854-482f-9c6b-c622dfae6753 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:52.725896 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-453e58d2-2854-482f-9c6b-c622dfae6753 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:52.726316 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-453e58d2-2854-482f-9c6b-c622dfae6753 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:52.726596 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-453e58d2-2854-482f-9c6b-c622dfae6753 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:52.737522 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-453e58d2-2854-482f-9c6b-c622dfae6753 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 21:19:52.743485 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-453e58d2-2854-482f-9c6b-c622dfae6753 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 29 21:19:52.743485 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:19:52.743485 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:19:52.743485 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:19:52.743485 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:19:52.743485 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:19:52.743485 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:19:52.743485 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:19:52.743485 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:19:52.743485 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:19:52.743485 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:19:52.743485 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:19:52.743485 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 21:19:52.743485 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 21:19:52.743485 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 21:19:52.743485 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 21:19:52.744333 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 21:19:52.744333 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 21:19:52.744333 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 21:19:52.744333 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 21:19:52.744333 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:19:52.744333 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:19:52.744333 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:19:52.744333 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:19:52.744333 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:19:52.744333 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:19:52.744333 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:19:52.744333 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:19:52.744333 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:19:52.744333 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:19:52.744333 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:19:52.744333 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:19:52.745530 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:19:52.745530 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:19:52.745530 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:19:52.745530 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:19:52.745530 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 21:19:52.745530 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 21:19:52.745530 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 21:19:52.745530 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 21:19:52.745530 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 21:19:52.745530 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 21:19:52.745530 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 29 21:19:52.745530 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:19:52.745530 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 73/144] 10.222.0.90 () {56 vars in 1099 bytes} [Mon Aug 29 21:19:52 2022] GET /identity/v3/roles/reader => generated 84 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:19:52.750445 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7314b668-bbd2-4420-affb-c28396c69556 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:52.876566 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7314b668-bbd2-4420-affb-c28396c69556 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:52.915246 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aacb8c01-aeaa-4649-9bc1-afd930f590da None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:52.915574 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aacb8c01-aeaa-4649-9bc1-afd930f590da None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:52.915796 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aacb8c01-aeaa-4649-9bc1-afd930f590da None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:52.917150 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 74/145] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:19:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 29 21:19:52.923320 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06df2f6c-8cca-4140-8993-7b2b915b96c9 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:52.923607 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06df2f6c-8cca-4140-8993-7b2b915b96c9 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:52.923882 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06df2f6c-8cca-4140-8993-7b2b915b96c9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:52.938262 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fb4ba73-8a53-4a5b-9493-e21baa34a6bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:52.938903 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb4ba73-8a53-4a5b-9493-e21baa34a6bd None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:52.939255 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb4ba73-8a53-4a5b-9493-e21baa34a6bd None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:52.939540 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb4ba73-8a53-4a5b-9493-e21baa34a6bd None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:52.940237 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fb4ba73-8a53-4a5b-9493-e21baa34a6bd None admin] RBAC: Adding query filter params (name=reader) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:19:52.940548 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fb4ba73-8a53-4a5b-9493-e21baa34a6bd None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:52.941997 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fb4ba73-8a53-4a5b-9493-e21baa34a6bd None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:52.952111 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 72/146] 10.222.0.90 () {56 vars in 1101 bytes} [Mon Aug 29 21:19:52 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:19:52.953981 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-06df2f6c-8cca-4140-8993-7b2b915b96c9 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:19:52.955207 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-06df2f6c-8cca-4140-8993-7b2b915b96c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:19:52.968044 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fb4ba73-8a53-4a5b-9493-e21baa34a6bd None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:53.021183 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:53.081074 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 75/147] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:19:52 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 21:19:53.129734 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fb4ba73-8a53-4a5b-9493-e21baa34a6bd None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:53.173501 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11d74ccb-c4ab-48a7-bf20-dac910f3d000 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:53.174180 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11d74ccb-c4ab-48a7-bf20-dac910f3d000 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:53.174468 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11d74ccb-c4ab-48a7-bf20-dac910f3d000 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:53.174753 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11d74ccb-c4ab-48a7-bf20-dac910f3d000 None admin] PATH_INFO: `/v3/users/b85a8c7cd85e4ae38ddbc5f2b5af87c5` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:53.178665 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11d74ccb-c4ab-48a7-bf20-dac910f3d000 None admin] RBAC: Authorizing `identity:get_user(user_id=b85a8c7cd85e4ae38ddbc5f2b5af87c5)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:53.180195 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11d74ccb-c4ab-48a7-bf20-dac910f3d000 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:53.181955 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 73/148] 10.222.0.90 () {56 vars in 1177 bytes} [Mon Aug 29 21:19:52 2022] GET /identity/v3/users/b85a8c7cd85e4ae38ddbc5f2b5af87c5 => generated 289 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:19:53.187681 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06df2f6c-8cca-4140-8993-7b2b915b96c9 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:53.211078 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:53.309064 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce088f4-4a06-4b4a-8e96-679d437ebd5b None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:53.309379 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce088f4-4a06-4b4a-8e96-679d437ebd5b None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:53.309613 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce088f4-4a06-4b4a-8e96-679d437ebd5b None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:53.311238 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 74/149] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:19:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:19:53.315844 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74e04c63-db78-4575-83b3-c96640df51c3 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:53.316126 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74e04c63-db78-4575-83b3-c96640df51c3 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:53.316400 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74e04c63-db78-4575-83b3-c96640df51c3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:53.322103 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06df2f6c-8cca-4140-8993-7b2b915b96c9 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:53.343364 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-74e04c63-db78-4575-83b3-c96640df51c3 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:19:53.344772 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-74e04c63-db78-4575-83b3-c96640df51c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:19:53.384086 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e0b373b-067c-40f6-817b-7731a6bd7230 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:53.384703 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e0b373b-067c-40f6-817b-7731a6bd7230 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:53.385002 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e0b373b-067c-40f6-817b-7731a6bd7230 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:53.385285 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e0b373b-067c-40f6-817b-7731a6bd7230 None admin] PATH_INFO: `/v3/projects/381cc34791e147329798cd454f99f622` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:53.387297 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e0b373b-067c-40f6-817b-7731a6bd7230 None admin] RBAC: Authorizing `identity:get_project(project_id=381cc34791e147329798cd454f99f622)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:53.388826 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e0b373b-067c-40f6-817b-7731a6bd7230 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:53.390554 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 76/150] 10.222.0.90 () {56 vars in 1186 bytes} [Mon Aug 29 21:19:53 2022] GET /identity/v3/projects/381cc34791e147329798cd454f99f622 => generated 298 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:19:53.396308 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e0b373b-067c-40f6-817b-7731a6bd7230 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:53.507535 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 75/151] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:19:53 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 29 21:19:53.519122 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:53.528767 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e0b373b-067c-40f6-817b-7731a6bd7230 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:53.577392 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43d25ab0-8ce8-4816-8133-1c93b1e5deb2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:53.578193 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d25ab0-8ce8-4816-8133-1c93b1e5deb2 None admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:53.578381 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d25ab0-8ce8-4816-8133-1c93b1e5deb2 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:53.578658 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d25ab0-8ce8-4816-8133-1c93b1e5deb2 None admin] PATH_INFO: `/v3/projects/381cc34791e147329798cd454f99f622/users/b85a8c7cd85e4ae38ddbc5f2b5af87c5/roles/414a14ad48a345f89fd1045f5d8fd16c` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:53.583887 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43d25ab0-8ce8-4816-8133-1c93b1e5deb2 None admin] RBAC: Authorizing `identity:create_grant(project_id=381cc34791e147329798cd454f99f622, user_id=b85a8c7cd85e4ae38ddbc5f2b5af87c5, role_id=414a14ad48a345f89fd1045f5d8fd16c)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:53.585343 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43d25ab0-8ce8-4816-8133-1c93b1e5deb2 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:53.603557 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 77/152] 10.222.0.90 () {58 vars in 1439 bytes} [Mon Aug 29 21:19:53 2022] PUT /identity/v3/projects/381cc34791e147329798cd454f99f622/users/b85a8c7cd85e4ae38ddbc5f2b5af87c5/roles/414a14ad48a345f89fd1045f5d8fd16c => generated 0 bytes in 207 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 21:19:53.694707 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74e04c63-db78-4575-83b3-c96640df51c3 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:53.734090 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:53.757237 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83ab302b-9abe-46a2-a91a-8ec5a4fc12ca None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:53.757583 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83ab302b-9abe-46a2-a91a-8ec5a4fc12ca None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:53.757851 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83ab302b-9abe-46a2-a91a-8ec5a4fc12ca None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:53.759486 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 78/153] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:19:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:19:53.765343 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58ac58cc-18bb-4369-88fb-68855229d884 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:53.765627 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58ac58cc-18bb-4369-88fb-68855229d884 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:53.765957 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58ac58cc-18bb-4369-88fb-68855229d884 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:53.802232 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-58ac58cc-18bb-4369-88fb-68855229d884 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:19:53.803462 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-58ac58cc-18bb-4369-88fb-68855229d884 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:19:53.847532 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74e04c63-db78-4575-83b3-c96640df51c3 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:53.931439 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d2107fe-685b-4c96-bd08-eb07786e1a91 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:53.932104 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d2107fe-685b-4c96-bd08-eb07786e1a91 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:53.932387 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d2107fe-685b-4c96-bd08-eb07786e1a91 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:53.932665 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d2107fe-685b-4c96-bd08-eb07786e1a91 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:53.951030 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 79/154] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:19:53 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 21:19:53.962734 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d2107fe-685b-4c96-bd08-eb07786e1a91 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 21:19:53.967727 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-1d2107fe-685b-4c96-bd08-eb07786e1a91 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 29 21:19:53.967727 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:19:53.967727 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:19:53.967727 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:19:53.967727 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:19:53.967727 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:19:53.967727 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:19:53.967727 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:19:53.967727 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:19:53.967727 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:19:53.967727 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:19:53.967727 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:19:53.967727 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 21:19:53.967727 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 21:19:53.967727 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 21:19:53.967727 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 21:19:53.968630 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 21:19:53.968630 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 21:19:53.968630 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 21:19:53.968630 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 21:19:53.968630 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:19:53.968630 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:19:53.968630 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:19:53.968630 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:19:53.968630 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:19:53.968630 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:19:53.968630 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:19:53.968630 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:19:53.968630 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:19:53.968630 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:19:53.968630 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:19:53.968630 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:19:53.969429 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:19:53.969429 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:19:53.969429 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:19:53.969429 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:19:53.969429 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 21:19:53.969429 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 21:19:53.969429 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 21:19:53.969429 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 21:19:53.969429 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 21:19:53.969429 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 21:19:53.969429 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 29 21:19:53.969429 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:19:53.969429 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 76/155] 10.222.0.90 () {56 vars in 1114 bytes} [Mon Aug 29 21:19:53 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 275 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:19:53.975795 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58ac58cc-18bb-4369-88fb-68855229d884 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:54.119997 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:54.142141 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d2107fe-685b-4c96-bd08-eb07786e1a91 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:54.148187 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58ac58cc-18bb-4369-88fb-68855229d884 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:54.197903 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4778484b-a677-44c0-89d3-6e93a6a13c32 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:54.198532 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4778484b-a677-44c0-89d3-6e93a6a13c32 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:54.198813 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4778484b-a677-44c0-89d3-6e93a6a13c32 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:54.199088 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4778484b-a677-44c0-89d3-6e93a6a13c32 None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:54.199769 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4778484b-a677-44c0-89d3-6e93a6a13c32 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:19:54.200072 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4778484b-a677-44c0-89d3-6e93a6a13c32 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:54.201515 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4778484b-a677-44c0-89d3-6e93a6a13c32 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:54.217544 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 80/156] 10.222.0.90 () {56 vars in 1111 bytes} [Mon Aug 29 21:19:53 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 242 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:19:54.224039 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4778484b-a677-44c0-89d3-6e93a6a13c32 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:54.382448 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:54.404017 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d2107fe-685b-4c96-bd08-eb07786e1a91 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:54.452550 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4778484b-a677-44c0-89d3-6e93a6a13c32 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:54.495386 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6e9e38f-6514-47f3-b538-122e49c73d6e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:54.496046 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6e9e38f-6514-47f3-b538-122e49c73d6e None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:54.496332 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6e9e38f-6514-47f3-b538-122e49c73d6e None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:54.496609 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6e9e38f-6514-47f3-b538-122e49c73d6e None admin] PATH_INFO: `/v3/roles/member` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:54.519111 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82e79230-e492-46a6-83da-6f09ac417da2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:54.520209 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82e79230-e492-46a6-83da-6f09ac417da2 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:54.521185 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82e79230-e492-46a6-83da-6f09ac417da2 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:54.521837 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82e79230-e492-46a6-83da-6f09ac417da2 None admin] PATH_INFO: `/v3/users/94fd78dbbf6e44308bd6e72458d6361f` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:54.522737 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6e9e38f-6514-47f3-b538-122e49c73d6e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 21:19:54.523463 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82e79230-e492-46a6-83da-6f09ac417da2 None admin] RBAC: Authorizing `identity:get_user(user_id=94fd78dbbf6e44308bd6e72458d6361f)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:54.525099 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82e79230-e492-46a6-83da-6f09ac417da2 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:54.526926 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 81/157] 10.222.0.90 () {56 vars in 1177 bytes} [Mon Aug 29 21:19:54 2022] GET /identity/v3/users/94fd78dbbf6e44308bd6e72458d6361f => generated 275 bytes in 303 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:19:54.527647 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-f6e9e38f-6514-47f3-b538-122e49c73d6e None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 29 21:19:54.527647 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:19:54.527647 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:19:54.527647 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:19:54.527647 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:19:54.527647 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:19:54.527647 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:19:54.527647 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:19:54.527647 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:19:54.527647 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:19:54.527647 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:19:54.527647 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:19:54.527647 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 21:19:54.527647 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 21:19:54.527647 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 21:19:54.527647 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 21:19:54.528662 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 21:19:54.528662 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 21:19:54.528662 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 21:19:54.528662 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 21:19:54.528662 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:19:54.528662 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:19:54.528662 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:19:54.528662 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:19:54.528662 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:19:54.528662 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:19:54.528662 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:19:54.528662 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:19:54.528662 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:19:54.528662 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:19:54.528662 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:19:54.528662 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:19:54.529550 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:19:54.529550 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:19:54.529550 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:19:54.529550 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:19:54.529550 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 21:19:54.529550 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 21:19:54.529550 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 21:19:54.529550 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 21:19:54.529550 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 21:19:54.529550 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 21:19:54.529550 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 29 21:19:54.529550 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:19:54.529550 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 77/158] 10.222.0.90 () {56 vars in 1099 bytes} [Mon Aug 29 21:19:54 2022] GET /identity/v3/roles/member => generated 84 bytes in 387 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 29 21:19:54.532888 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82e79230-e492-46a6-83da-6f09ac417da2 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:54.534238 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6e9e38f-6514-47f3-b538-122e49c73d6e None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:54.545658 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:54.705436 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82e79230-e492-46a6-83da-6f09ac417da2 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:54.711766 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6e9e38f-6514-47f3-b538-122e49c73d6e None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:54.774695 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-769eb5c8-5a36-46c1-928c-37ef6c936b9d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:54.775331 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-769eb5c8-5a36-46c1-928c-37ef6c936b9d None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:54.775612 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-769eb5c8-5a36-46c1-928c-37ef6c936b9d None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:54.775885 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-769eb5c8-5a36-46c1-928c-37ef6c936b9d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:54.786762 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6781c095-436f-4f2a-8652-13e3897076a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:54.787414 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6781c095-436f-4f2a-8652-13e3897076a1 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:54.787699 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6781c095-436f-4f2a-8652-13e3897076a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:54.787975 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6781c095-436f-4f2a-8652-13e3897076a1 None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:54.789090 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6781c095-436f-4f2a-8652-13e3897076a1 None admin] RBAC: Adding query filter params (name=member) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:19:54.789399 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6781c095-436f-4f2a-8652-13e3897076a1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:54.791450 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6781c095-436f-4f2a-8652-13e3897076a1 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:54.808061 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 78/159] 10.222.0.90 () {56 vars in 1101 bytes} [Mon Aug 29 21:19:54 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 274 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:19:54.810324 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6781c095-436f-4f2a-8652-13e3897076a1 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:54.839803 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:54.898299 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-769eb5c8-5a36-46c1-928c-37ef6c936b9d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:54.899793 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-769eb5c8-5a36-46c1-928c-37ef6c936b9d None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:54.907957 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:55.002123 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6781c095-436f-4f2a-8652-13e3897076a1 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:55.068887 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 82/160] 10.222.0.90 () {58 vars in 1282 bytes} [Mon Aug 29 21:19:54 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 536 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 29 21:19:55.070202 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24c47047-ace2-46d3-a57e-8db400f8f2cc None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:55.070490 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24c47047-ace2-46d3-a57e-8db400f8f2cc None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:55.070767 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24c47047-ace2-46d3-a57e-8db400f8f2cc None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:55.072171 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 83/161] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:19:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:19:55.075152 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0142026-38b1-42b0-ae2f-f5aadd7dd75f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:55.075799 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0142026-38b1-42b0-ae2f-f5aadd7dd75f None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:55.076083 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0142026-38b1-42b0-ae2f-f5aadd7dd75f None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:55.076361 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0142026-38b1-42b0-ae2f-f5aadd7dd75f None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:55.079005 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24c47047-ace2-46d3-a57e-8db400f8f2cc None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:55.099440 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0142026-38b1-42b0-ae2f-f5aadd7dd75f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 21:19:55.105337 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-b0142026-38b1-42b0-ae2f-f5aadd7dd75f None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 21:19:55.105337 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:19:55.105337 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:19:55.105337 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:19:55.105337 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:19:55.105337 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:19:55.105337 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:19:55.105337 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:19:55.105337 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:19:55.105337 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:19:55.105337 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:19:55.105337 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:19:55.105337 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 21:19:55.105337 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 21:19:55.105337 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 21:19:55.105337 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 21:19:55.106268 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 21:19:55.106268 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 21:19:55.106268 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 21:19:55.106268 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 21:19:55.106268 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:19:55.106268 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:19:55.106268 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:19:55.106268 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:19:55.106268 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:19:55.106268 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:19:55.106268 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:19:55.106268 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:19:55.106268 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:19:55.106268 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:19:55.106268 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:19:55.106268 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:19:55.107106 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:19:55.107106 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:19:55.107106 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:19:55.107106 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:19:55.107106 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 21:19:55.107106 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 21:19:55.107106 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 21:19:55.107106 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 21:19:55.107106 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 21:19:55.107106 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 21:19:55.107106 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 21:19:55.107106 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:19:55.107106 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 79/162] 10.222.0.90 () {56 vars in 1096 bytes} [Mon Aug 29 21:19:54 2022] GET /identity/v3/roles/admin => generated 83 bytes in 298 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:19:55.107777 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0142026-38b1-42b0-ae2f-f5aadd7dd75f None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:55.202177 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:55.230279 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0142026-38b1-42b0-ae2f-f5aadd7dd75f None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:55.267055 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-084d31bd-703d-491c-a788-219c24aae0db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:55.267634 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-084d31bd-703d-491c-a788-219c24aae0db None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:55.267921 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-084d31bd-703d-491c-a788-219c24aae0db None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:55.268201 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-084d31bd-703d-491c-a788-219c24aae0db None admin] PATH_INFO: `/v3/projects/381cc34791e147329798cd454f99f622` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:55.269348 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-084d31bd-703d-491c-a788-219c24aae0db None admin] RBAC: Authorizing `identity:get_project(project_id=381cc34791e147329798cd454f99f622)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:55.271053 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24c47047-ace2-46d3-a57e-8db400f8f2cc None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:55.271429 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-084d31bd-703d-491c-a788-219c24aae0db None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:55.274526 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 80/163] 10.222.0.90 () {56 vars in 1186 bytes} [Mon Aug 29 21:19:55 2022] GET /identity/v3/projects/381cc34791e147329798cd454f99f622 => generated 298 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:19:55.275728 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3eebb40-568f-43a4-ac88-81e9463ab8b0 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:55.276012 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3eebb40-568f-43a4-ac88-81e9463ab8b0 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:55.276289 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3eebb40-568f-43a4-ac88-81e9463ab8b0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:55.282196 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:55.301019 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-b3eebb40-568f-43a4-ac88-81e9463ab8b0 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:19:55.302672 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-b3eebb40-568f-43a4-ac88-81e9463ab8b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:19:55.376490 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 81/164] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:19:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 21:19:55.377465 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3eebb40-568f-43a4-ac88-81e9463ab8b0 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:55.386536 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6111b6c2-0b11-45ec-87a4-b58a524307e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:55.387253 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6111b6c2-0b11-45ec-87a4-b58a524307e9 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:55.387538 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6111b6c2-0b11-45ec-87a4-b58a524307e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:55.387810 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6111b6c2-0b11-45ec-87a4-b58a524307e9 None admin] PATH_INFO: `/v3/users/94fd78dbbf6e44308bd6e72458d6361f` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:55.390301 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6111b6c2-0b11-45ec-87a4-b58a524307e9 None admin] RBAC: Authorizing `identity:get_user(user_id=94fd78dbbf6e44308bd6e72458d6361f)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:55.391879 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6111b6c2-0b11-45ec-87a4-b58a524307e9 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:55.393498 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 84/165] 10.222.0.90 () {56 vars in 1177 bytes} [Mon Aug 29 21:19:55 2022] GET /identity/v3/users/94fd78dbbf6e44308bd6e72458d6361f => generated 275 bytes in 321 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:19:55.394292 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6111b6c2-0b11-45ec-87a4-b58a524307e9 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:55.458936 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:55.493440 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3eebb40-568f-43a4-ac88-81e9463ab8b0 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:55.531961 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8faa2756-f20e-444e-aefb-179880ceee16 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:55.533011 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8faa2756-f20e-444e-aefb-179880ceee16 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:55.533309 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8faa2756-f20e-444e-aefb-179880ceee16 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:55.533585 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8faa2756-f20e-444e-aefb-179880ceee16 None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:55.534289 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8faa2756-f20e-444e-aefb-179880ceee16 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:19:55.534597 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8faa2756-f20e-444e-aefb-179880ceee16 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:55.536694 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8faa2756-f20e-444e-aefb-179880ceee16 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:55.545407 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 82/166] 10.222.0.90 () {56 vars in 1099 bytes} [Mon Aug 29 21:19:55 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:19:55.546246 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8faa2756-f20e-444e-aefb-179880ceee16 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:55.668096 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6111b6c2-0b11-45ec-87a4-b58a524307e9 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:55.680675 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8faa2756-f20e-444e-aefb-179880ceee16 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:55.712519 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:55.716436 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d8f241e-8a6a-41e6-a3db-7a56bee1056c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:55.717066 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d8f241e-8a6a-41e6-a3db-7a56bee1056c None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:55.717354 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d8f241e-8a6a-41e6-a3db-7a56bee1056c None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:55.717634 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d8f241e-8a6a-41e6-a3db-7a56bee1056c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:55.733339 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-452a2a23-29b3-419a-a4c7-d3106a95f3f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:55.734022 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-452a2a23-29b3-419a-a4c7-d3106a95f3f1 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:55.734288 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-452a2a23-29b3-419a-a4c7-d3106a95f3f1 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:55.734561 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-452a2a23-29b3-419a-a4c7-d3106a95f3f1 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:55.735309 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-452a2a23-29b3-419a-a4c7-d3106a95f3f1 None admin] RBAC: Adding query filter params (role.id=9745586777bf473bab7d887bf25bf236, scope.project.id=381cc34791e147329798cd454f99f622, user.id=94fd78dbbf6e44308bd6e72458d6361f) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:19:55.735619 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-452a2a23-29b3-419a-a4c7-d3106a95f3f1 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:55.737099 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-452a2a23-29b3-419a-a4c7-d3106a95f3f1 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:55.746240 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 85/167] 10.222.0.90 () {56 vars in 1374 bytes} [Mon Aug 29 21:19:55 2022] GET /identity/v3/role_assignments?user.id=94fd78dbbf6e44308bd6e72458d6361f&scope.project.id=381cc34791e147329798cd454f99f622&role.id=9745586777bf473bab7d887bf25bf236 => generated 608 bytes in 352 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:19:55.747074 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-452a2a23-29b3-419a-a4c7-d3106a95f3f1 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:55.833997 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d8f241e-8a6a-41e6-a3db-7a56bee1056c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:55.839697 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d8f241e-8a6a-41e6-a3db-7a56bee1056c None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:55.911079 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:55.985438 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 83/168] 10.222.0.90 () {58 vars in 1282 bytes} [Mon Aug 29 21:19:55 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 439 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 29 21:19:56.016786 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d8f241e-8a6a-41e6-a3db-7a56bee1056c None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:56.041169 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-452a2a23-29b3-419a-a4c7-d3106a95f3f1 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:56.063999 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:56.094587 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6f59c3d-2a6b-4435-9a1d-53cf9a9ffcb4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:56.095155 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f59c3d-2a6b-4435-9a1d-53cf9a9ffcb4 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:56.095440 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f59c3d-2a6b-4435-9a1d-53cf9a9ffcb4 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:56.095715 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f59c3d-2a6b-4435-9a1d-53cf9a9ffcb4 None admin] PATH_INFO: `/v3/users/97a0b159abf44cd59fe5a2b7bde28adf` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:56.097139 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6f59c3d-2a6b-4435-9a1d-53cf9a9ffcb4 None admin] RBAC: Authorizing `identity:get_user(user_id=97a0b159abf44cd59fe5a2b7bde28adf)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:56.098692 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6f59c3d-2a6b-4435-9a1d-53cf9a9ffcb4 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:56.100800 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 86/169] 10.222.0.90 () {56 vars in 1177 bytes} [Mon Aug 29 21:19:55 2022] GET /identity/v3/users/97a0b159abf44cd59fe5a2b7bde28adf => generated 247 bytes in 354 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:19:56.106183 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6f59c3d-2a6b-4435-9a1d-53cf9a9ffcb4 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:56.181160 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d8f241e-8a6a-41e6-a3db-7a56bee1056c None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:56.250632 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a454365e-47d4-425d-9fec-8340326828b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:56.251481 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a454365e-47d4-425d-9fec-8340326828b1 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:56.251738 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a454365e-47d4-425d-9fec-8340326828b1 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:56.251961 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a454365e-47d4-425d-9fec-8340326828b1 None admin] PATH_INFO: `/v3/projects/381cc34791e147329798cd454f99f622` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:56.253275 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a454365e-47d4-425d-9fec-8340326828b1 None admin] RBAC: Authorizing `identity:get_project(project_id=381cc34791e147329798cd454f99f622)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:56.255639 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a454365e-47d4-425d-9fec-8340326828b1 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:56.258823 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 84/170] 10.222.0.90 () {56 vars in 1186 bytes} [Mon Aug 29 21:19:56 2022] GET /identity/v3/projects/381cc34791e147329798cd454f99f622 => generated 298 bytes in 242 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:19:56.259735 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a454365e-47d4-425d-9fec-8340326828b1 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:56.293756 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6f59c3d-2a6b-4435-9a1d-53cf9a9ffcb4 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:56.304926 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:56.352270 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84ab763f-b3bd-41da-81c4-bb1cf5b57acb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:56.352818 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84ab763f-b3bd-41da-81c4-bb1cf5b57acb None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:56.353169 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84ab763f-b3bd-41da-81c4-bb1cf5b57acb None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:56.353450 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84ab763f-b3bd-41da-81c4-bb1cf5b57acb None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:56.473103 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:56.480062 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a454365e-47d4-425d-9fec-8340326828b1 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:56.499063 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84ab763f-b3bd-41da-81c4-bb1cf5b57acb None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:56.500886 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84ab763f-b3bd-41da-81c4-bb1cf5b57acb None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:56.552673 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e59db81-934f-4a9d-aa67-efcfda17373a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:56.553929 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e59db81-934f-4a9d-aa67-efcfda17373a None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:56.555100 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e59db81-934f-4a9d-aa67-efcfda17373a None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:56.556821 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e59db81-934f-4a9d-aa67-efcfda17373a None admin] PATH_INFO: `/v3/roles/member` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:56.580797 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e59db81-934f-4a9d-aa67-efcfda17373a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 21:19:56.590133 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-1e59db81-934f-4a9d-aa67-efcfda17373a None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 29 21:19:56.590133 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:19:56.590133 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:19:56.590133 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:19:56.590133 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:19:56.590133 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:19:56.590133 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:19:56.590133 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:19:56.590133 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:19:56.590133 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:19:56.590133 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:19:56.590133 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:19:56.590133 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 21:19:56.590133 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 21:19:56.590133 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 21:19:56.590133 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 21:19:56.591156 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 21:19:56.591156 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 21:19:56.591156 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 21:19:56.591156 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 21:19:56.591156 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:19:56.591156 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:19:56.591156 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:19:56.591156 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:19:56.591156 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:19:56.591156 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:19:56.591156 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:19:56.591156 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:19:56.591156 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:19:56.591156 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:19:56.591156 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:19:56.591156 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:19:56.591959 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:19:56.591959 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:19:56.591959 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:19:56.591959 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:19:56.591959 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 21:19:56.591959 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 21:19:56.591959 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 21:19:56.591959 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 21:19:56.591959 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 21:19:56.591959 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 21:19:56.591959 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 29 21:19:56.591959 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:19:56.595203 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 85/171] 10.222.0.90 () {56 vars in 1099 bytes} [Mon Aug 29 21:19:56 2022] GET /identity/v3/roles/member => generated 84 bytes in 336 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:19:56.599743 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e59db81-934f-4a9d-aa67-efcfda17373a None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:56.696312 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 87/172] 10.222.0.90 () {58 vars in 1282 bytes} [Mon Aug 29 21:19:56 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 590 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Aug 29 21:19:56.700009 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84ab763f-b3bd-41da-81c4-bb1cf5b57acb None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:56.710376 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:56.829127 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e59db81-934f-4a9d-aa67-efcfda17373a None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:56.878001 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff7c1d06-5394-4503-b679-8fb4c15713cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:56.879027 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff7c1d06-5394-4503-b679-8fb4c15713cf None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:56.880317 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff7c1d06-5394-4503-b679-8fb4c15713cf None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:56.880911 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff7c1d06-5394-4503-b679-8fb4c15713cf None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:56.882778 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff7c1d06-5394-4503-b679-8fb4c15713cf None admin] RBAC: Adding query filter params (role.id=02e242d5520740d4b77a5f4008c7eb57, scope.project.id=381cc34791e147329798cd454f99f622, user.id=94fd78dbbf6e44308bd6e72458d6361f) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:19:56.883912 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff7c1d06-5394-4503-b679-8fb4c15713cf None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:56.885824 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff7c1d06-5394-4503-b679-8fb4c15713cf None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:56.890704 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84ab763f-b3bd-41da-81c4-bb1cf5b57acb None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:56.896095 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 86/173] 10.222.0.90 () {56 vars in 1374 bytes} [Mon Aug 29 21:19:56 2022] GET /identity/v3/role_assignments?user.id=94fd78dbbf6e44308bd6e72458d6361f&scope.project.id=381cc34791e147329798cd454f99f622&role.id=02e242d5520740d4b77a5f4008c7eb57 => generated 608 bytes in 300 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:19:56.897734 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff7c1d06-5394-4503-b679-8fb4c15713cf None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:56.955503 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:56.966459 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed921877-8d8c-4bbf-abc2-95bf034e4a11 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:56.966859 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed921877-8d8c-4bbf-abc2-95bf034e4a11 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:56.967142 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed921877-8d8c-4bbf-abc2-95bf034e4a11 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:56.967411 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed921877-8d8c-4bbf-abc2-95bf034e4a11 None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:56.968109 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed921877-8d8c-4bbf-abc2-95bf034e4a11 None admin] RBAC: Adding query filter params (name=member) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:19:56.968432 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed921877-8d8c-4bbf-abc2-95bf034e4a11 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:56.969837 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed921877-8d8c-4bbf-abc2-95bf034e4a11 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:56.981905 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 88/174] 10.222.0.90 () {56 vars in 1101 bytes} [Mon Aug 29 21:19:56 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 285 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:19:56.983425 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10387c79-3b69-40d3-8f90-8f458e65e584 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:56.983708 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10387c79-3b69-40d3-8f90-8f458e65e584 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:56.983999 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10387c79-3b69-40d3-8f90-8f458e65e584 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:56.994044 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 89/175] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:19:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:19:56.994044 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb32c291-f0c6-4001-8e39-d01fa6695ff6 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:56.994044 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb32c291-f0c6-4001-8e39-d01fa6695ff6 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:56.994044 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb32c291-f0c6-4001-8e39-d01fa6695ff6 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:56.994044 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 90/176] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:19:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:19:56.994044 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bb1b3a8-22d8-4600-adfa-4ff7fbf70e37 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:56.994044 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bb1b3a8-22d8-4600-adfa-4ff7fbf70e37 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:56.994621 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bb1b3a8-22d8-4600-adfa-4ff7fbf70e37 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:56.994621 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 91/177] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:19:56 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:19:56.996886 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bb1b3a8-22d8-4600-adfa-4ff7fbf70e37 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:57.078050 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff7c1d06-5394-4503-b679-8fb4c15713cf None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:57.083900 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:57.116169 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d792de49-07e8-42be-ba63-5dd74fdb37bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:57.117206 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d792de49-07e8-42be-ba63-5dd74fdb37bd None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:57.117716 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d792de49-07e8-42be-ba63-5dd74fdb37bd None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:57.118178 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d792de49-07e8-42be-ba63-5dd74fdb37bd None admin] PATH_INFO: `/v3/projects/381cc34791e147329798cd454f99f622` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:57.119503 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d792de49-07e8-42be-ba63-5dd74fdb37bd None admin] RBAC: Authorizing `identity:get_project(project_id=381cc34791e147329798cd454f99f622)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:57.121116 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d792de49-07e8-42be-ba63-5dd74fdb37bd None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:57.122961 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 87/178] 10.222.0.90 () {56 vars in 1186 bytes} [Mon Aug 29 21:19:56 2022] GET /identity/v3/projects/381cc34791e147329798cd454f99f622 => generated 298 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:19:57.124084 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-199f55c8-416e-46cf-b696-23336ae4ea65 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:57.124427 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-199f55c8-416e-46cf-b696-23336ae4ea65 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:57.124703 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-199f55c8-416e-46cf-b696-23336ae4ea65 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:57.127186 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bb1b3a8-22d8-4600-adfa-4ff7fbf70e37 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:57.143500 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-199f55c8-416e-46cf-b696-23336ae4ea65 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:19:57.144695 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-199f55c8-416e-46cf-b696-23336ae4ea65 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:19:57.167185 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a47b88d-f9d6-43e7-86a5-847a7139d7cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:57.167831 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a47b88d-f9d6-43e7-86a5-847a7139d7cb None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:57.168127 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a47b88d-f9d6-43e7-86a5-847a7139d7cb None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:57.168383 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a47b88d-f9d6-43e7-86a5-847a7139d7cb None admin] PATH_INFO: `/v3/users/94fd78dbbf6e44308bd6e72458d6361f` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:57.169514 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a47b88d-f9d6-43e7-86a5-847a7139d7cb None admin] RBAC: Authorizing `identity:get_user(user_id=94fd78dbbf6e44308bd6e72458d6361f)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:57.171006 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a47b88d-f9d6-43e7-86a5-847a7139d7cb None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:57.172584 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 92/179] 10.222.0.90 () {56 vars in 1177 bytes} [Mon Aug 29 21:19:56 2022] GET /identity/v3/users/94fd78dbbf6e44308bd6e72458d6361f => generated 275 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:19:57.173678 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c1ebae5-abb7-424b-886b-0e6116d84480 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:57.173978 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c1ebae5-abb7-424b-886b-0e6116d84480 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:57.174250 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c1ebae5-abb7-424b-886b-0e6116d84480 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:57.194947 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-8c1ebae5-abb7-424b-886b-0e6116d84480 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:19:57.196547 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-8c1ebae5-abb7-424b-886b-0e6116d84480 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:19:57.222367 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 88/180] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:19:57 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 21:19:57.223972 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84475535-e420-4b28-8671-eba01f648439 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:57.224263 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84475535-e420-4b28-8671-eba01f648439 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:57.224545 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84475535-e420-4b28-8671-eba01f648439 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:57.246466 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-84475535-e420-4b28-8671-eba01f648439 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:19:57.247900 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-84475535-e420-4b28-8671-eba01f648439 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:19:57.267273 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:57.277348 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 93/181] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:19:57 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 21:19:57.278394 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c1ebae5-abb7-424b-886b-0e6116d84480 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:57.337257 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 89/182] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:19:57 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 21:19:57.341909 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84475535-e420-4b28-8671-eba01f648439 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:57.374489 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:57.460575 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c1ebae5-abb7-424b-886b-0e6116d84480 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:57.533216 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb2f112a-f65e-483f-92e1-72392db0eed4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:57.533859 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb2f112a-f65e-483f-92e1-72392db0eed4 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:57.534161 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb2f112a-f65e-483f-92e1-72392db0eed4 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:57.534423 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb2f112a-f65e-483f-92e1-72392db0eed4 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:57.535165 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb2f112a-f65e-483f-92e1-72392db0eed4 None admin] RBAC: Adding query filter params (role.id=a6e562cdc96c4be691d638990f8403fd, scope.project.id=381cc34791e147329798cd454f99f622, user.id=97a0b159abf44cd59fe5a2b7bde28adf) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:19:57.535474 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb2f112a-f65e-483f-92e1-72392db0eed4 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:57.537662 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb2f112a-f65e-483f-92e1-72392db0eed4 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:57.547931 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 94/183] 10.222.0.90 () {56 vars in 1374 bytes} [Mon Aug 29 21:19:57 2022] GET /identity/v3/role_assignments?user.id=97a0b159abf44cd59fe5a2b7bde28adf&scope.project.id=381cc34791e147329798cd454f99f622&role.id=a6e562cdc96c4be691d638990f8403fd => generated 608 bytes in 270 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:19:57.564803 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84475535-e420-4b28-8671-eba01f648439 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:57.568630 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb2f112a-f65e-483f-92e1-72392db0eed4 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:57.624362 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47889119-2653-4530-84ac-80928df0f3fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:57.624994 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47889119-2653-4530-84ac-80928df0f3fb None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:57.625270 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47889119-2653-4530-84ac-80928df0f3fb None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:57.625543 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47889119-2653-4530-84ac-80928df0f3fb None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:57.661737 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:57.709978 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb2f112a-f65e-483f-92e1-72392db0eed4 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:57.714859 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47889119-2653-4530-84ac-80928df0f3fb None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:57.716347 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47889119-2653-4530-84ac-80928df0f3fb None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:57.748467 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:57.756869 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e1bf023-d23d-4be3-83f9-4d2c9b9193a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:57.757549 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e1bf023-d23d-4be3-83f9-4d2c9b9193a5 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:57.757907 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e1bf023-d23d-4be3-83f9-4d2c9b9193a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:57.758121 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e1bf023-d23d-4be3-83f9-4d2c9b9193a5 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:57.771624 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e1bf023-d23d-4be3-83f9-4d2c9b9193a5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 21:19:57.776903 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-4e1bf023-d23d-4be3-83f9-4d2c9b9193a5 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 21:19:57.776903 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:19:57.776903 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:19:57.776903 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:19:57.776903 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:19:57.776903 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:19:57.776903 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:19:57.776903 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:19:57.776903 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:19:57.776903 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:19:57.776903 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:19:57.776903 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:19:57.776903 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 21:19:57.776903 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 21:19:57.776903 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 21:19:57.776903 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 21:19:57.778009 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 21:19:57.778009 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 21:19:57.778009 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 21:19:57.778009 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 21:19:57.778009 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:19:57.778009 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:19:57.778009 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:19:57.778009 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:19:57.778009 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:19:57.778009 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:19:57.778009 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:19:57.778009 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:19:57.778009 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:19:57.778009 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:19:57.778009 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:19:57.778009 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:19:57.778838 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:19:57.778838 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:19:57.778838 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:19:57.778838 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:19:57.778838 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 21:19:57.778838 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 21:19:57.778838 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 21:19:57.778838 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 21:19:57.778838 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 21:19:57.778838 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 21:19:57.778838 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 21:19:57.778838 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:19:57.780434 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 95/184] 10.222.0.90 () {56 vars in 1096 bytes} [Mon Aug 29 21:19:57 2022] GET /identity/v3/roles/admin => generated 83 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 29 21:19:57.780434 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e1bf023-d23d-4be3-83f9-4d2c9b9193a5 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:57.837108 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 90/185] 10.222.0.90 () {58 vars in 1282 bytes} [Mon Aug 29 21:19:57 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 499 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 29 21:19:57.838745 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c815f4a8-3170-4e81-b6bc-c9f95777bd70 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:57.839027 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c815f4a8-3170-4e81-b6bc-c9f95777bd70 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:57.839303 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c815f4a8-3170-4e81-b6bc-c9f95777bd70 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:57.840611 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 91/186] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:19:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:19:57.841379 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c815f4a8-3170-4e81-b6bc-c9f95777bd70 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:57.897499 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:57.899000 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e1bf023-d23d-4be3-83f9-4d2c9b9193a5 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:57.934026 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a445bff3-360e-4afa-b524-ff7d15f7620b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:57.934652 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a445bff3-360e-4afa-b524-ff7d15f7620b None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:57.934928 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a445bff3-360e-4afa-b524-ff7d15f7620b None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:57.935196 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a445bff3-360e-4afa-b524-ff7d15f7620b None admin] PATH_INFO: `/v3/domains/default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:57.936290 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a445bff3-360e-4afa-b524-ff7d15f7620b None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:57.937730 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a445bff3-360e-4afa-b524-ff7d15f7620b None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:57.939324 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 96/187] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:19:57 2022] GET /identity/v3/domains/default => generated 199 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:19:57.941913 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a445bff3-360e-4afa-b524-ff7d15f7620b None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:57.948427 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c815f4a8-3170-4e81-b6bc-c9f95777bd70 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:57.982705 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83533ebc-d073-404d-a34d-bb58da0f1fce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:57.983343 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83533ebc-d073-404d-a34d-bb58da0f1fce None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:57.983623 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83533ebc-d073-404d-a34d-bb58da0f1fce None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:57.983897 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83533ebc-d073-404d-a34d-bb58da0f1fce None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:57.995100 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83533ebc-d073-404d-a34d-bb58da0f1fce None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 21:19:57.997792 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:58.000083 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-83533ebc-d073-404d-a34d-bb58da0f1fce None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 29 21:19:58.000083 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:19:58.000083 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:19:58.000083 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:19:58.000083 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:19:58.000083 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:19:58.000083 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:19:58.000083 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:19:58.000083 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:19:58.000083 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:19:58.000083 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:19:58.000083 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:19:58.000083 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 21:19:58.000083 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 21:19:58.000083 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 21:19:58.000083 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 21:19:58.001090 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 21:19:58.001090 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 21:19:58.001090 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 21:19:58.001090 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 21:19:58.001090 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:19:58.001090 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:19:58.001090 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:19:58.001090 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:19:58.001090 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:19:58.001090 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:19:58.001090 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:19:58.001090 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:19:58.001090 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:19:58.001090 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:19:58.001090 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:19:58.001090 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:19:58.002100 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:19:58.002100 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:19:58.002100 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:19:58.002100 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:19:58.002100 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 21:19:58.002100 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 21:19:58.002100 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 21:19:58.002100 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 21:19:58.002100 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 21:19:58.002100 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 21:19:58.002100 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 29 21:19:58.002100 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:19:58.002100 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 92/188] 10.222.0.90 () {56 vars in 1114 bytes} [Mon Aug 29 21:19:57 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:19:58.003355 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c75201a6-b426-46e3-b726-2a9de9c4819e None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:58.003355 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c75201a6-b426-46e3-b726-2a9de9c4819e None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:58.003596 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c75201a6-b426-46e3-b726-2a9de9c4819e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:58.021655 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-c75201a6-b426-46e3-b726-2a9de9c4819e None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:19:58.022889 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-c75201a6-b426-46e3-b726-2a9de9c4819e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:19:58.042636 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a445bff3-360e-4afa-b524-ff7d15f7620b None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:58.076803 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b887df6b-4dad-4834-9a92-422c04c14a1c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:58.077404 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b887df6b-4dad-4834-9a92-422c04c14a1c None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:58.077682 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b887df6b-4dad-4834-9a92-422c04c14a1c None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:58.077980 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b887df6b-4dad-4834-9a92-422c04c14a1c None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:58.078673 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b887df6b-4dad-4834-9a92-422c04c14a1c None admin] RBAC: Adding query filter params (name=admin) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:19:58.079076 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b887df6b-4dad-4834-9a92-422c04c14a1c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:58.080457 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b887df6b-4dad-4834-9a92-422c04c14a1c None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:58.093200 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 97/189] 10.222.0.90 () {56 vars in 1099 bytes} [Mon Aug 29 21:19:57 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:19:58.093200 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b887df6b-4dad-4834-9a92-422c04c14a1c None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:58.094396 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 93/190] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:19:58 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 21:19:58.095206 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c75201a6-b426-46e3-b726-2a9de9c4819e None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:58.141062 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:58.203857 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b887df6b-4dad-4834-9a92-422c04c14a1c None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:58.204652 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c75201a6-b426-46e3-b726-2a9de9c4819e None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:58.227245 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:58.245621 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-001ecd0c-5541-4ab0-8dc2-48af13ee39b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:58.246382 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-001ecd0c-5541-4ab0-8dc2-48af13ee39b1 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:58.246660 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-001ecd0c-5541-4ab0-8dc2-48af13ee39b1 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:58.246934 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-001ecd0c-5541-4ab0-8dc2-48af13ee39b1 None admin] PATH_INFO: `/v3/projects/c7d7a59f097c47d89a7e42b80adb5b50` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:58.248085 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-001ecd0c-5541-4ab0-8dc2-48af13ee39b1 None admin] RBAC: Authorizing `identity:get_project(project_id=c7d7a59f097c47d89a7e42b80adb5b50)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:58.248588 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6826b793-34b3-4561-85f2-edd5585064d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:58.249210 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6826b793-34b3-4561-85f2-edd5585064d1 None admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:58.249493 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6826b793-34b3-4561-85f2-edd5585064d1 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:58.249586 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-001ecd0c-5541-4ab0-8dc2-48af13ee39b1 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:58.249805 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6826b793-34b3-4561-85f2-edd5585064d1 None admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:58.250504 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6826b793-34b3-4561-85f2-edd5585064d1 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:58.256585 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 98/191] 10.222.0.90 () {56 vars in 1186 bytes} [Mon Aug 29 21:19:58 2022] GET /identity/v3/projects/c7d7a59f097c47d89a7e42b80adb5b50 => generated 312 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:19:58.256585 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-001ecd0c-5541-4ab0-8dc2-48af13ee39b1 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:58.256801 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6826b793-34b3-4561-85f2-edd5585064d1 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:58.277643 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 94/192] 10.222.0.90 () {60 vars in 1132 bytes} [Mon Aug 29 21:19:58 2022] POST /identity/v3/users => generated 297 bytes in 183 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:19:58.277643 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6826b793-34b3-4561-85f2-edd5585064d1 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:58.377228 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-001ecd0c-5541-4ab0-8dc2-48af13ee39b1 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:58.417273 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ccb6eb5-826b-48a7-ab65-78346e93026f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:58.417921 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ccb6eb5-826b-48a7-ab65-78346e93026f None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:58.418199 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ccb6eb5-826b-48a7-ab65-78346e93026f None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:58.418469 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ccb6eb5-826b-48a7-ab65-78346e93026f None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:58.419142 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ccb6eb5-826b-48a7-ab65-78346e93026f None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:19:58.419473 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ccb6eb5-826b-48a7-ab65-78346e93026f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:58.420958 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ccb6eb5-826b-48a7-ab65-78346e93026f None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:58.435927 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 99/193] 10.222.0.90 () {56 vars in 1111 bytes} [Mon Aug 29 21:19:58 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:19:58.435927 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ccb6eb5-826b-48a7-ab65-78346e93026f None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:58.436224 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:58.441626 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6826b793-34b3-4561-85f2-edd5585064d1 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:58.484848 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f1becff-d0fc-477d-97c6-97cbfc3139db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:58.485534 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f1becff-d0fc-477d-97c6-97cbfc3139db None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:58.486282 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f1becff-d0fc-477d-97c6-97cbfc3139db None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:58.486282 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f1becff-d0fc-477d-97c6-97cbfc3139db None admin] PATH_INFO: `/v3/users/cce3ca7fa2c74842a2105931d0c0aa39` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:58.489990 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:58.503012 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f1becff-d0fc-477d-97c6-97cbfc3139db None admin] RBAC: Authorizing `identity:get_user(user_id=cce3ca7fa2c74842a2105931d0c0aa39)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:58.504843 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f1becff-d0fc-477d-97c6-97cbfc3139db None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:58.506510 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 95/194] 10.222.0.90 () {56 vars in 1177 bytes} [Mon Aug 29 21:19:58 2022] GET /identity/v3/users/cce3ca7fa2c74842a2105931d0c0aa39 => generated 283 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:19:58.507308 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f1becff-d0fc-477d-97c6-97cbfc3139db None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:58.564528 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ccb6eb5-826b-48a7-ab65-78346e93026f None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:58.607336 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d69516f2-49cf-4fb1-9e21-438fd0e69d87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:58.607958 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d69516f2-49cf-4fb1-9e21-438fd0e69d87 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:58.608242 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d69516f2-49cf-4fb1-9e21-438fd0e69d87 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:58.608519 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d69516f2-49cf-4fb1-9e21-438fd0e69d87 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:58.609259 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d69516f2-49cf-4fb1-9e21-438fd0e69d87 None admin] RBAC: Adding query filter params (role.id=02e242d5520740d4b77a5f4008c7eb57, scope.project.id=c7d7a59f097c47d89a7e42b80adb5b50, user.id=94fd78dbbf6e44308bd6e72458d6361f) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:19:58.609586 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d69516f2-49cf-4fb1-9e21-438fd0e69d87 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:58.611079 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d69516f2-49cf-4fb1-9e21-438fd0e69d87 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:58.618658 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 100/195] 10.222.0.90 () {56 vars in 1374 bytes} [Mon Aug 29 21:19:58 2022] GET /identity/v3/role_assignments?user.id=94fd78dbbf6e44308bd6e72458d6361f&scope.project.id=c7d7a59f097c47d89a7e42b80adb5b50&role.id=02e242d5520740d4b77a5f4008c7eb57 => generated 608 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:19:58.619411 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d69516f2-49cf-4fb1-9e21-438fd0e69d87 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:58.643380 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f1becff-d0fc-477d-97c6-97cbfc3139db None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:58.696443 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31791580-ed3d-4f86-9105-ef93c587c4cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:58.697105 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31791580-ed3d-4f86-9105-ef93c587c4cf None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:58.697419 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31791580-ed3d-4f86-9105-ef93c587c4cf None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:58.697707 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31791580-ed3d-4f86-9105-ef93c587c4cf None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:58.715676 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31791580-ed3d-4f86-9105-ef93c587c4cf None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 21:19:58.723217 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-31791580-ed3d-4f86-9105-ef93c587c4cf None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 29 21:19:58.723217 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:19:58.723217 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:19:58.723217 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:19:58.723217 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:19:58.723217 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:19:58.723217 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:19:58.723217 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:19:58.723217 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:19:58.723217 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:19:58.723217 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:19:58.723217 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:19:58.723217 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 21:19:58.723217 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 21:19:58.723217 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 21:19:58.723217 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 21:19:58.724862 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 21:19:58.724862 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 21:19:58.724862 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 21:19:58.724862 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 21:19:58.724862 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:19:58.724862 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:19:58.724862 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:19:58.724862 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:19:58.724862 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:19:58.724862 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:19:58.724862 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:19:58.724862 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:19:58.724862 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:19:58.724862 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:19:58.724862 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:19:58.724862 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:19:58.726422 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:19:58.726422 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:19:58.726422 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:19:58.726422 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:19:58.726422 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 21:19:58.726422 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 21:19:58.726422 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 21:19:58.726422 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 21:19:58.726422 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 21:19:58.726422 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 21:19:58.726422 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 29 21:19:58.726422 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:19:58.726422 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 96/196] 10.222.0.90 () {56 vars in 1099 bytes} [Mon Aug 29 21:19:58 2022] GET /identity/v3/roles/reader => generated 84 bytes in 218 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:19:58.726422 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31791580-ed3d-4f86-9105-ef93c587c4cf None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:58.756512 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:58.767421 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d69516f2-49cf-4fb1-9e21-438fd0e69d87 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:58.791431 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:58.821261 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29503772-2a7d-4ef4-b0eb-461256068049 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:58.822031 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29503772-2a7d-4ef4-b0eb-461256068049 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:58.822317 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29503772-2a7d-4ef4-b0eb-461256068049 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:58.822587 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29503772-2a7d-4ef4-b0eb-461256068049 None admin] PATH_INFO: `/v3/users/cce3ca7fa2c74842a2105931d0c0aa39` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:58.823895 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29503772-2a7d-4ef4-b0eb-461256068049 None admin] RBAC: Authorizing `identity:get_user(user_id=cce3ca7fa2c74842a2105931d0c0aa39)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:58.825879 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29503772-2a7d-4ef4-b0eb-461256068049 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:58.831632 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 101/197] 10.222.0.90 () {56 vars in 1177 bytes} [Mon Aug 29 21:19:58 2022] GET /identity/v3/users/cce3ca7fa2c74842a2105931d0c0aa39 => generated 283 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:19:58.831632 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29503772-2a7d-4ef4-b0eb-461256068049 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:58.879725 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31791580-ed3d-4f86-9105-ef93c587c4cf None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:58.949839 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ff23658-c79e-4ff5-90d6-d74b5d49198c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:58.950705 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ff23658-c79e-4ff5-90d6-d74b5d49198c None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:58.951133 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ff23658-c79e-4ff5-90d6-d74b5d49198c None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:58.951572 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ff23658-c79e-4ff5-90d6-d74b5d49198c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:58.978600 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29503772-2a7d-4ef4-b0eb-461256068049 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:59.019783 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1423366-f99f-448f-a9e7-5b436381efbd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:59.020606 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1423366-f99f-448f-a9e7-5b436381efbd None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:59.020890 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1423366-f99f-448f-a9e7-5b436381efbd None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:59.021158 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1423366-f99f-448f-a9e7-5b436381efbd None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:59.021966 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1423366-f99f-448f-a9e7-5b436381efbd None admin] RBAC: Adding query filter params (name=reader) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:19:59.022266 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1423366-f99f-448f-a9e7-5b436381efbd None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:59.023700 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1423366-f99f-448f-a9e7-5b436381efbd None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:59.032710 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 102/198] 10.222.0.90 () {56 vars in 1101 bytes} [Mon Aug 29 21:19:58 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:19:59.035423 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1423366-f99f-448f-a9e7-5b436381efbd None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:59.045289 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ff23658-c79e-4ff5-90d6-d74b5d49198c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:59.047228 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ff23658-c79e-4ff5-90d6-d74b5d49198c None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:59.073526 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:59.095116 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:59.178568 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1423366-f99f-448f-a9e7-5b436381efbd None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:59.187296 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 97/199] 10.222.0.90 () {58 vars in 1282 bytes} [Mon Aug 29 21:19:58 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 462 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 29 21:19:59.188594 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ff23658-c79e-4ff5-90d6-d74b5d49198c None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:59.228910 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05f0909c-f553-4d2e-8248-e12bff3643f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:59.229465 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f0909c-f553-4d2e-8248-e12bff3643f3 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:59.229741 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f0909c-f553-4d2e-8248-e12bff3643f3 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:59.230040 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f0909c-f553-4d2e-8248-e12bff3643f3 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:59.316560 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05f0909c-f553-4d2e-8248-e12bff3643f3 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:59.318160 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05f0909c-f553-4d2e-8248-e12bff3643f3 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:59.327245 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ff23658-c79e-4ff5-90d6-d74b5d49198c None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:59.354066 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:59.367054 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:59.383214 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b1de9c8-6ae7-4f0d-8213-a4cb617ba1c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:59.384298 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b1de9c8-6ae7-4f0d-8213-a4cb617ba1c7 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:59.384760 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b1de9c8-6ae7-4f0d-8213-a4cb617ba1c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:59.385490 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b1de9c8-6ae7-4f0d-8213-a4cb617ba1c7 None admin] PATH_INFO: `/v3/users/b85a8c7cd85e4ae38ddbc5f2b5af87c5` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:59.387132 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b1de9c8-6ae7-4f0d-8213-a4cb617ba1c7 None admin] RBAC: Authorizing `identity:get_user(user_id=b85a8c7cd85e4ae38ddbc5f2b5af87c5)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:59.388936 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b1de9c8-6ae7-4f0d-8213-a4cb617ba1c7 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:59.391674 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 98/200] 10.222.0.90 () {56 vars in 1177 bytes} [Mon Aug 29 21:19:59 2022] GET /identity/v3/users/b85a8c7cd85e4ae38ddbc5f2b5af87c5 => generated 289 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:19:59.392943 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b1de9c8-6ae7-4f0d-8213-a4cb617ba1c7 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:59.432261 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 103/201] 10.222.0.90 () {58 vars in 1282 bytes} [Mon Aug 29 21:19:59 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 395 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 29 21:19:59.432261 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05f0909c-f553-4d2e-8248-e12bff3643f3 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:59.509074 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b1de9c8-6ae7-4f0d-8213-a4cb617ba1c7 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:59.533987 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05f0909c-f553-4d2e-8248-e12bff3643f3 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:59.546713 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e4d051d-5a53-4042-9040-7a1fb4aabdeb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:59.547614 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e4d051d-5a53-4042-9040-7a1fb4aabdeb None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:59.548073 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e4d051d-5a53-4042-9040-7a1fb4aabdeb None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:59.548618 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e4d051d-5a53-4042-9040-7a1fb4aabdeb None admin] PATH_INFO: `/v3/projects/f6d8dbb619904d8889100044bad1bef6` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:59.550078 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e4d051d-5a53-4042-9040-7a1fb4aabdeb None admin] RBAC: Authorizing `identity:get_project(project_id=f6d8dbb619904d8889100044bad1bef6)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:59.551836 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e4d051d-5a53-4042-9040-7a1fb4aabdeb None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:59.553913 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 99/202] 10.222.0.90 () {56 vars in 1186 bytes} [Mon Aug 29 21:19:59 2022] GET /identity/v3/projects/f6d8dbb619904d8889100044bad1bef6 => generated 302 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:19:59.555089 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e4d051d-5a53-4042-9040-7a1fb4aabdeb None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:59.571361 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2acd1c2-1102-4fbb-a144-870f9a4af50c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:59.572056 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2acd1c2-1102-4fbb-a144-870f9a4af50c None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:59.572447 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2acd1c2-1102-4fbb-a144-870f9a4af50c None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:59.572787 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2acd1c2-1102-4fbb-a144-870f9a4af50c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:59.588351 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:59.619312 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:59.650537 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2acd1c2-1102-4fbb-a144-870f9a4af50c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:59.652161 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2acd1c2-1102-4fbb-a144-870f9a4af50c None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:59.674412 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e4d051d-5a53-4042-9040-7a1fb4aabdeb None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:59.714449 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21ce7940-ef83-465b-a164-ad8abf454d67 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:59.715385 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21ce7940-ef83-465b-a164-ad8abf454d67 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:59.715940 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21ce7940-ef83-465b-a164-ad8abf454d67 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:59.716424 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21ce7940-ef83-465b-a164-ad8abf454d67 None admin] PATH_INFO: `/v3/projects/f6d8dbb619904d8889100044bad1bef6` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:59.717769 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21ce7940-ef83-465b-a164-ad8abf454d67 None admin] RBAC: Authorizing `identity:get_project(project_id=f6d8dbb619904d8889100044bad1bef6)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:59.719426 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21ce7940-ef83-465b-a164-ad8abf454d67 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:59.726938 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 100/203] 10.222.0.90 () {56 vars in 1186 bytes} [Mon Aug 29 21:19:59 2022] GET /identity/v3/projects/f6d8dbb619904d8889100044bad1bef6 => generated 302 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:19:59.728009 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21ce7940-ef83-465b-a164-ad8abf454d67 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:59.759581 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 104/204] 10.222.0.90 () {58 vars in 1282 bytes} [Mon Aug 29 21:19:59 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 331 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 29 21:19:59.760257 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2acd1c2-1102-4fbb-a144-870f9a4af50c None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:59.807950 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:59.845121 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21ce7940-ef83-465b-a164-ad8abf454d67 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:59.865456 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2acd1c2-1102-4fbb-a144-870f9a4af50c None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:19:59.871941 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:19:59.889745 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c869578f-b78a-4719-8bb6-0e34d4bbec7e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:59.890759 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c869578f-b78a-4719-8bb6-0e34d4bbec7e None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:59.891323 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c869578f-b78a-4719-8bb6-0e34d4bbec7e None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:59.891774 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c869578f-b78a-4719-8bb6-0e34d4bbec7e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:59.892712 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c869578f-b78a-4719-8bb6-0e34d4bbec7e None admin] RBAC: Adding query filter params (role.id=a6e562cdc96c4be691d638990f8403fd, scope.project.id=f6d8dbb619904d8889100044bad1bef6, user.id=cce3ca7fa2c74842a2105931d0c0aa39) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:19:59.893286 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c869578f-b78a-4719-8bb6-0e34d4bbec7e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:59.895200 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c869578f-b78a-4719-8bb6-0e34d4bbec7e None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:59.901718 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 101/205] 10.222.0.90 () {56 vars in 1374 bytes} [Mon Aug 29 21:19:59 2022] GET /identity/v3/role_assignments?user.id=cce3ca7fa2c74842a2105931d0c0aa39&scope.project.id=f6d8dbb619904d8889100044bad1bef6&role.id=a6e562cdc96c4be691d638990f8403fd => generated 259 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:19:59.902829 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c869578f-b78a-4719-8bb6-0e34d4bbec7e None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:19:59.907355 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a9d56b8-ef3b-423c-a3be-8da0f905adf4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:19:59.907355 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a9d56b8-ef3b-423c-a3be-8da0f905adf4 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:19:59.907605 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a9d56b8-ef3b-423c-a3be-8da0f905adf4 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:19:59.907915 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a9d56b8-ef3b-423c-a3be-8da0f905adf4 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:19:59.908724 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a9d56b8-ef3b-423c-a3be-8da0f905adf4 None admin] RBAC: Adding query filter params (role.id=9745586777bf473bab7d887bf25bf236, scope.project.id=f6d8dbb619904d8889100044bad1bef6, user.id=cce3ca7fa2c74842a2105931d0c0aa39) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:19:59.909098 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a9d56b8-ef3b-423c-a3be-8da0f905adf4 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:19:59.911294 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a9d56b8-ef3b-423c-a3be-8da0f905adf4 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:19:59.924009 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 105/206] 10.222.0.90 () {56 vars in 1374 bytes} [Mon Aug 29 21:19:59 2022] GET /identity/v3/role_assignments?user.id=cce3ca7fa2c74842a2105931d0c0aa39&scope.project.id=f6d8dbb619904d8889100044bad1bef6&role.id=9745586777bf473bab7d887bf25bf236 => generated 259 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:00.075186 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c869578f-b78a-4719-8bb6-0e34d4bbec7e None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:00.125186 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-274b46bc-cad0-4fd1-8876-d90363e4fe04 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:00.126299 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-274b46bc-cad0-4fd1-8876-d90363e4fe04 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:00.126789 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-274b46bc-cad0-4fd1-8876-d90363e4fe04 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:00.127271 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-274b46bc-cad0-4fd1-8876-d90363e4fe04 None admin] PATH_INFO: `/v3/projects/381cc34791e147329798cd454f99f622` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:00.128706 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-274b46bc-cad0-4fd1-8876-d90363e4fe04 None admin] RBAC: Authorizing `identity:get_project(project_id=381cc34791e147329798cd454f99f622)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:00.130647 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-274b46bc-cad0-4fd1-8876-d90363e4fe04 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:00.132705 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 102/207] 10.222.0.90 () {56 vars in 1186 bytes} [Mon Aug 29 21:19:59 2022] GET /identity/v3/projects/381cc34791e147329798cd454f99f622 => generated 298 bytes in 230 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:00.138908 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a9d56b8-ef3b-423c-a3be-8da0f905adf4 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:00.264526 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a9d56b8-ef3b-423c-a3be-8da0f905adf4 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:00.274497 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:00.302679 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e32fc0bf-9e7f-4ef2-b2c4-dfa4ed7be0a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:00.303299 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e32fc0bf-9e7f-4ef2-b2c4-dfa4ed7be0a2 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:00.303576 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e32fc0bf-9e7f-4ef2-b2c4-dfa4ed7be0a2 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:00.303839 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e32fc0bf-9e7f-4ef2-b2c4-dfa4ed7be0a2 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:00.304568 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e32fc0bf-9e7f-4ef2-b2c4-dfa4ed7be0a2 None admin] RBAC: Adding query filter params (role.id=414a14ad48a345f89fd1045f5d8fd16c, scope.project.id=381cc34791e147329798cd454f99f622, user.id=b85a8c7cd85e4ae38ddbc5f2b5af87c5) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:00.304868 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e32fc0bf-9e7f-4ef2-b2c4-dfa4ed7be0a2 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:00.306352 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e32fc0bf-9e7f-4ef2-b2c4-dfa4ed7be0a2 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:00.312848 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 106/208] 10.222.0.90 () {56 vars in 1374 bytes} [Mon Aug 29 21:20:00 2022] GET /identity/v3/role_assignments?user.id=b85a8c7cd85e4ae38ddbc5f2b5af87c5&scope.project.id=381cc34791e147329798cd454f99f622&role.id=414a14ad48a345f89fd1045f5d8fd16c => generated 608 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:00.541545 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbe7f5b0-8801-46ce-9d70-e33e41e11265 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:00.542079 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbe7f5b0-8801-46ce-9d70-e33e41e11265 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:00.542156 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbe7f5b0-8801-46ce-9d70-e33e41e11265 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:00.543743 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 103/209] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:20:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:20:00.548883 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d122a46-87cf-4d66-979d-b7612b9c724d None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:00.549170 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d122a46-87cf-4d66-979d-b7612b9c724d None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:00.549442 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d122a46-87cf-4d66-979d-b7612b9c724d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:00.571574 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-1d122a46-87cf-4d66-979d-b7612b9c724d None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:20:00.572785 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-1d122a46-87cf-4d66-979d-b7612b9c724d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:20:00.645180 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 107/210] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:20:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 29 21:20:00.680733 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e363fbf8-d93b-4de1-889b-ceeb2f462b2b None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:00.681032 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e363fbf8-d93b-4de1-889b-ceeb2f462b2b None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:00.681282 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e363fbf8-d93b-4de1-889b-ceeb2f462b2b None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:00.682666 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 104/211] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:20:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:20:00.688032 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65d1fd09-c1d4-4b42-9fb4-9cac362948d5 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:00.688312 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65d1fd09-c1d4-4b42-9fb4-9cac362948d5 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:00.688580 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65d1fd09-c1d4-4b42-9fb4-9cac362948d5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:00.706976 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-65d1fd09-c1d4-4b42-9fb4-9cac362948d5 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:20:00.708710 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-65d1fd09-c1d4-4b42-9fb4-9cac362948d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:20:00.785433 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 108/212] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:20:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 21:20:00.954947 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e363fbf8-d93b-4de1-889b-ceeb2f462b2b None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:01.016856 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:01.070141 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e363fbf8-d93b-4de1-889b-ceeb2f462b2b None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:01.080936 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65d1fd09-c1d4-4b42-9fb4-9cac362948d5 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:01.108499 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cc2912c-cdb5-4f04-acb4-1c1ec61b9c39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:01.109658 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cc2912c-cdb5-4f04-acb4-1c1ec61b9c39 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:01.110177 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cc2912c-cdb5-4f04-acb4-1c1ec61b9c39 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:01.110651 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cc2912c-cdb5-4f04-acb4-1c1ec61b9c39 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:01.112748 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:01.127450 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cc2912c-cdb5-4f04-acb4-1c1ec61b9c39 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 21:20:01.133489 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-2cc2912c-cdb5-4f04-acb4-1c1ec61b9c39 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 29 21:20:01.133489 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:01.133489 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:01.133489 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:01.133489 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:01.133489 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:01.133489 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:01.133489 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:01.133489 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:01.133489 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:01.133489 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:01.133489 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:01.133489 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 21:20:01.133489 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 21:20:01.133489 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 21:20:01.133489 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 21:20:01.134417 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 21:20:01.134417 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 21:20:01.134417 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 21:20:01.134417 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 21:20:01.134417 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:01.134417 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:01.134417 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:01.134417 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:01.134417 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:01.134417 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:01.134417 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:01.134417 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:01.134417 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:01.134417 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:01.134417 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:01.134417 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:01.135234 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:01.135234 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:01.135234 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:01.135234 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:01.135234 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 21:20:01.135234 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 21:20:01.135234 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 21:20:01.135234 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 21:20:01.135234 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 21:20:01.135234 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 21:20:01.135234 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 29 21:20:01.135234 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:20:01.142292 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 105/213] 10.222.0.90 () {56 vars in 1099 bytes} [Mon Aug 29 21:20:00 2022] GET /identity/v3/roles/member => generated 84 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:20:01.143834 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cc2912c-cdb5-4f04-acb4-1c1ec61b9c39 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:01.210047 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65d1fd09-c1d4-4b42-9fb4-9cac362948d5 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:01.260599 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56ab16f3-fdb6-4e08-bd71-92e6b5010d42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:01.261250 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ab16f3-fdb6-4e08-bd71-92e6b5010d42 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:01.261531 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ab16f3-fdb6-4e08-bd71-92e6b5010d42 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:01.261834 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ab16f3-fdb6-4e08-bd71-92e6b5010d42 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:01.262960 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56ab16f3-fdb6-4e08-bd71-92e6b5010d42 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:01.263588 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cc2912c-cdb5-4f04-acb4-1c1ec61b9c39 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:01.264409 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56ab16f3-fdb6-4e08-bd71-92e6b5010d42 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:01.266482 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 109/214] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:20:01 2022] GET /identity/v3/domains/default => generated 199 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:20:01.274694 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56ab16f3-fdb6-4e08-bd71-92e6b5010d42 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:01.281971 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:01.305880 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73186440-97cf-4f73-a9e9-613888bffaca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:01.306892 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73186440-97cf-4f73-a9e9-613888bffaca None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:01.307371 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73186440-97cf-4f73-a9e9-613888bffaca None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:01.307806 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73186440-97cf-4f73-a9e9-613888bffaca None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:01.308664 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73186440-97cf-4f73-a9e9-613888bffaca None admin] RBAC: Adding query filter params (name=member) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:01.309125 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73186440-97cf-4f73-a9e9-613888bffaca None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:01.310862 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73186440-97cf-4f73-a9e9-613888bffaca None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:01.323052 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 106/215] 10.222.0.90 () {56 vars in 1101 bytes} [Mon Aug 29 21:20:01 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:20:01.328674 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73186440-97cf-4f73-a9e9-613888bffaca None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:01.403536 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:01.409427 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56ab16f3-fdb6-4e08-bd71-92e6b5010d42 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:01.439609 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73186440-97cf-4f73-a9e9-613888bffaca None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:01.456760 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f8e102a-8fb0-46b2-9605-3509e265715d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:01.457958 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f8e102a-8fb0-46b2-9605-3509e265715d None admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:01.458831 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f8e102a-8fb0-46b2-9605-3509e265715d None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:01.459247 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f8e102a-8fb0-46b2-9605-3509e265715d None admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:01.460129 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f8e102a-8fb0-46b2-9605-3509e265715d None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:01.462305 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f8e102a-8fb0-46b2-9605-3509e265715d None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:01.487131 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c2cba22-8bad-4a83-af85-3a9bd533d37f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:01.488009 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c2cba22-8bad-4a83-af85-3a9bd533d37f None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:01.488461 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c2cba22-8bad-4a83-af85-3a9bd533d37f None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:01.489390 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c2cba22-8bad-4a83-af85-3a9bd533d37f None admin] PATH_INFO: `/v3/users/1e04ff7645fa4bae8e6f0ddbf7d824ab` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:01.509826 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 110/216] 10.222.0.90 () {60 vars in 1132 bytes} [Mon Aug 29 21:20:01 2022] POST /identity/v3/users => generated 297 bytes in 232 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:20:01.521735 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c2cba22-8bad-4a83-af85-3a9bd533d37f None admin] RBAC: Authorizing `identity:get_user(user_id=1e04ff7645fa4bae8e6f0ddbf7d824ab)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:01.523417 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c2cba22-8bad-4a83-af85-3a9bd533d37f None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:01.525293 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 107/217] 10.222.0.90 () {56 vars in 1177 bytes} [Mon Aug 29 21:20:01 2022] GET /identity/v3/users/1e04ff7645fa4bae8e6f0ddbf7d824ab => generated 297 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:20:01.531514 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f8e102a-8fb0-46b2-9605-3509e265715d None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:01.646717 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f8e102a-8fb0-46b2-9605-3509e265715d None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:01.684970 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e3081e9-1b25-468c-a8bb-568c48818497 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:01.685967 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e3081e9-1b25-468c-a8bb-568c48818497 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:01.686467 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e3081e9-1b25-468c-a8bb-568c48818497 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:01.686950 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e3081e9-1b25-468c-a8bb-568c48818497 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:01.715258 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:01.764942 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e3081e9-1b25-468c-a8bb-568c48818497 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:01.766511 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e3081e9-1b25-468c-a8bb-568c48818497 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:01.882369 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 111/218] 10.222.0.90 () {58 vars in 1282 bytes} [Mon Aug 29 21:20:01 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 351 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 29 21:20:01.888450 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c2cba22-8bad-4a83-af85-3a9bd533d37f None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:01.923373 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:02.006370 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c2cba22-8bad-4a83-af85-3a9bd533d37f None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:02.060287 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5a2fa6c-704e-43f4-9ccc-a7e4cfa30dc6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:02.060998 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5a2fa6c-704e-43f4-9ccc-a7e4cfa30dc6 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:02.061276 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5a2fa6c-704e-43f4-9ccc-a7e4cfa30dc6 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:02.061550 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5a2fa6c-704e-43f4-9ccc-a7e4cfa30dc6 None admin] PATH_INFO: `/v3/projects/f6d8dbb619904d8889100044bad1bef6` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:02.062758 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5a2fa6c-704e-43f4-9ccc-a7e4cfa30dc6 None admin] RBAC: Authorizing `identity:get_project(project_id=f6d8dbb619904d8889100044bad1bef6)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:02.064192 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5a2fa6c-704e-43f4-9ccc-a7e4cfa30dc6 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:02.065918 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 108/219] 10.222.0.90 () {56 vars in 1186 bytes} [Mon Aug 29 21:20:01 2022] GET /identity/v3/projects/f6d8dbb619904d8889100044bad1bef6 => generated 302 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:02.076777 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e3081e9-1b25-468c-a8bb-568c48818497 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:02.190193 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:02.201952 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41639d13-96c0-4585-977d-ccf892b16f54 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:02.202326 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41639d13-96c0-4585-977d-ccf892b16f54 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:02.202499 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41639d13-96c0-4585-977d-ccf892b16f54 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:02.204052 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 109/220] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:20:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 29 21:20:02.209076 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-114593ce-b57d-4d3f-b9b3-c173f278fbb8 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:02.209404 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-114593ce-b57d-4d3f-b9b3-c173f278fbb8 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:02.209645 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-114593ce-b57d-4d3f-b9b3-c173f278fbb8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:02.247950 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-114593ce-b57d-4d3f-b9b3-c173f278fbb8 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:20:02.249292 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-114593ce-b57d-4d3f-b9b3-c173f278fbb8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:20:02.273911 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e3081e9-1b25-468c-a8bb-568c48818497 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:02.355373 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2321ebc7-8ef9-439b-939c-34520cd90cf6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:02.355992 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2321ebc7-8ef9-439b-939c-34520cd90cf6 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:02.356271 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2321ebc7-8ef9-439b-939c-34520cd90cf6 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:02.356544 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2321ebc7-8ef9-439b-939c-34520cd90cf6 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:02.357283 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2321ebc7-8ef9-439b-939c-34520cd90cf6 None admin] RBAC: Adding query filter params (role.id=02e242d5520740d4b77a5f4008c7eb57, scope.project.id=f6d8dbb619904d8889100044bad1bef6, user.id=1e04ff7645fa4bae8e6f0ddbf7d824ab) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:02.357587 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2321ebc7-8ef9-439b-939c-34520cd90cf6 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:02.359068 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2321ebc7-8ef9-439b-939c-34520cd90cf6 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:02.373827 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 112/221] 10.222.0.90 () {56 vars in 1374 bytes} [Mon Aug 29 21:20:02 2022] GET /identity/v3/role_assignments?user.id=1e04ff7645fa4bae8e6f0ddbf7d824ab&scope.project.id=f6d8dbb619904d8889100044bad1bef6&role.id=02e242d5520740d4b77a5f4008c7eb57 => generated 259 bytes in 299 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:02.378841 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-030b62a0-efb1-4fe6-9075-52d33aea7d8c None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:02.385817 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-030b62a0-efb1-4fe6-9075-52d33aea7d8c None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:02.385817 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-030b62a0-efb1-4fe6-9075-52d33aea7d8c None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:02.388745 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 113/222] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:20:02 2022] GET /identity => generated 270 bytes in 14 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:20:02.391798 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55826f66-9625-402d-a05e-a2216cf34537 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:02.393189 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55826f66-9625-402d-a05e-a2216cf34537 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:02.393189 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55826f66-9625-402d-a05e-a2216cf34537 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:02.408410 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 110/223] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:20:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 200 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 29 21:20:02.459763 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-55826f66-9625-402d-a05e-a2216cf34537 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:20:02.461737 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-55826f66-9625-402d-a05e-a2216cf34537 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:20:02.680064 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 114/224] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:20:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 290 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 21:20:02.880258 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-114593ce-b57d-4d3f-b9b3-c173f278fbb8 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:02.911894 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:02.992943 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-114593ce-b57d-4d3f-b9b3-c173f278fbb8 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:03.028549 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcd63ea0-314f-47f5-9957-1c154d646f14 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:03.029189 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd63ea0-314f-47f5-9957-1c154d646f14 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:03.029470 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd63ea0-314f-47f5-9957-1c154d646f14 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:03.029826 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd63ea0-314f-47f5-9957-1c154d646f14 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:03.040607 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcd63ea0-314f-47f5-9957-1c154d646f14 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 21:20:03.045458 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-bcd63ea0-314f-47f5-9957-1c154d646f14 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 21:20:03.045458 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:03.045458 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:03.045458 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:03.045458 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:03.045458 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:03.045458 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:03.045458 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:03.045458 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:03.045458 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:03.045458 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:03.045458 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:03.045458 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 21:20:03.045458 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 21:20:03.045458 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 21:20:03.045458 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 21:20:03.047080 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 21:20:03.047080 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 21:20:03.047080 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 21:20:03.047080 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 21:20:03.047080 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:03.047080 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:03.047080 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:03.047080 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:03.047080 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:03.047080 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:03.047080 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:03.047080 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:03.047080 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:03.047080 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:03.047080 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:03.047080 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:03.048641 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:03.048641 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:03.048641 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:03.048641 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:03.048641 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 21:20:03.048641 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 21:20:03.048641 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 21:20:03.048641 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 21:20:03.048641 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 21:20:03.048641 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 21:20:03.048641 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 21:20:03.048641 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:20:03.048641 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 111/225] 10.222.0.90 () {56 vars in 1096 bytes} [Mon Aug 29 21:20:02 2022] GET /identity/v3/roles/admin => generated 83 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 29 21:20:03.059485 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55826f66-9625-402d-a05e-a2216cf34537 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:03.134877 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:03.185766 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55826f66-9625-402d-a05e-a2216cf34537 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:03.225107 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-258fc68d-e1a4-4106-b58d-0725c08d5929 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:03.225742 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-258fc68d-e1a4-4106-b58d-0725c08d5929 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:03.226049 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-258fc68d-e1a4-4106-b58d-0725c08d5929 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:03.226325 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-258fc68d-e1a4-4106-b58d-0725c08d5929 None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:03.226589 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcd63ea0-314f-47f5-9957-1c154d646f14 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:03.227004 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-258fc68d-e1a4-4106-b58d-0725c08d5929 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:03.227305 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-258fc68d-e1a4-4106-b58d-0725c08d5929 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:03.228690 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-258fc68d-e1a4-4106-b58d-0725c08d5929 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:03.237724 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 115/226] 10.222.0.90 () {56 vars in 1099 bytes} [Mon Aug 29 21:20:03 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:03.250824 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-258fc68d-e1a4-4106-b58d-0725c08d5929 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:03.362120 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-258fc68d-e1a4-4106-b58d-0725c08d5929 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:03.370316 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:03.381331 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcd63ea0-314f-47f5-9957-1c154d646f14 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:03.385820 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:03.408967 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23afc8de-01b8-4cd1-975c-1a20181731ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:03.409726 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23afc8de-01b8-4cd1-975c-1a20181731ee None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:03.410124 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23afc8de-01b8-4cd1-975c-1a20181731ee None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:03.410490 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23afc8de-01b8-4cd1-975c-1a20181731ee None admin] PATH_INFO: `/v3/users/cce3ca7fa2c74842a2105931d0c0aa39` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:03.412528 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23afc8de-01b8-4cd1-975c-1a20181731ee None admin] RBAC: Authorizing `identity:get_user(user_id=cce3ca7fa2c74842a2105931d0c0aa39)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:03.414084 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23afc8de-01b8-4cd1-975c-1a20181731ee None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:03.415766 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 116/227] 10.222.0.90 () {56 vars in 1177 bytes} [Mon Aug 29 21:20:03 2022] GET /identity/v3/users/cce3ca7fa2c74842a2105931d0c0aa39 => generated 283 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:20:03.421197 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23afc8de-01b8-4cd1-975c-1a20181731ee None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:03.505000 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-267fe089-426f-45a3-913a-ecec521e2006 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:03.506174 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-267fe089-426f-45a3-913a-ecec521e2006 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:03.506659 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-267fe089-426f-45a3-913a-ecec521e2006 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:03.507126 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-267fe089-426f-45a3-913a-ecec521e2006 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:03.525477 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23afc8de-01b8-4cd1-975c-1a20181731ee None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:03.542719 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-267fe089-426f-45a3-913a-ecec521e2006 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 21:20:03.547584 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-267fe089-426f-45a3-913a-ecec521e2006 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 29 21:20:03.547584 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:03.547584 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:03.547584 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:03.547584 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:03.547584 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:03.547584 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:03.547584 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:03.547584 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:03.547584 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:03.547584 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:03.547584 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:03.547584 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 21:20:03.547584 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 21:20:03.547584 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 21:20:03.547584 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 21:20:03.549227 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 21:20:03.549227 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 21:20:03.549227 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 21:20:03.549227 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 21:20:03.549227 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:03.549227 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:03.549227 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:03.549227 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:03.549227 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:03.549227 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:03.549227 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:03.549227 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:03.549227 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:03.549227 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:03.549227 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:03.549227 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:03.551328 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:03.551328 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:03.551328 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:03.551328 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:03.551328 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 21:20:03.551328 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 21:20:03.551328 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 21:20:03.551328 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 21:20:03.551328 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 21:20:03.551328 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 21:20:03.551328 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 29 21:20:03.551328 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:20:03.551328 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 112/228] 10.222.0.90 () {56 vars in 1114 bytes} [Mon Aug 29 21:20:03 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 323 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:20:03.556969 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-267fe089-426f-45a3-913a-ecec521e2006 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:03.586107 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13e458db-66d7-459c-bf7a-73fda03a3b8f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:03.586749 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13e458db-66d7-459c-bf7a-73fda03a3b8f None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:03.587046 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13e458db-66d7-459c-bf7a-73fda03a3b8f None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:03.587284 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13e458db-66d7-459c-bf7a-73fda03a3b8f None admin] PATH_INFO: `/v3/projects/f6d8dbb619904d8889100044bad1bef6` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:03.589138 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13e458db-66d7-459c-bf7a-73fda03a3b8f None admin] RBAC: Authorizing `identity:get_project(project_id=f6d8dbb619904d8889100044bad1bef6)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:03.590607 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13e458db-66d7-459c-bf7a-73fda03a3b8f None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:03.592250 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 117/229] 10.222.0.90 () {56 vars in 1186 bytes} [Mon Aug 29 21:20:03 2022] GET /identity/v3/projects/f6d8dbb619904d8889100044bad1bef6 => generated 302 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:20:03.600089 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13e458db-66d7-459c-bf7a-73fda03a3b8f None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:03.671390 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:03.736389 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13e458db-66d7-459c-bf7a-73fda03a3b8f None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:03.773399 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c87af5c-fd13-4208-bd06-e43313d90d3e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:03.774014 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c87af5c-fd13-4208-bd06-e43313d90d3e None admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:03.774295 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c87af5c-fd13-4208-bd06-e43313d90d3e None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:03.774568 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c87af5c-fd13-4208-bd06-e43313d90d3e None admin] PATH_INFO: `/v3/projects/f6d8dbb619904d8889100044bad1bef6/users/cce3ca7fa2c74842a2105931d0c0aa39/roles/a6e562cdc96c4be691d638990f8403fd` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:03.776546 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c87af5c-fd13-4208-bd06-e43313d90d3e None admin] RBAC: Authorizing `identity:create_grant(project_id=f6d8dbb619904d8889100044bad1bef6, user_id=cce3ca7fa2c74842a2105931d0c0aa39, role_id=a6e562cdc96c4be691d638990f8403fd)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:03.780066 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c87af5c-fd13-4208-bd06-e43313d90d3e None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:03.794728 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 118/230] 10.222.0.90 () {58 vars in 1439 bytes} [Mon Aug 29 21:20:03 2022] PUT /identity/v3/projects/f6d8dbb619904d8889100044bad1bef6/users/cce3ca7fa2c74842a2105931d0c0aa39/roles/a6e562cdc96c4be691d638990f8403fd => generated 0 bytes in 195 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:20:03.852442 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-267fe089-426f-45a3-913a-ecec521e2006 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:03.956335 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-225f02d4-6e5d-4ab9-aa02-ef71f968c452 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:03.958057 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-225f02d4-6e5d-4ab9-aa02-ef71f968c452 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:03.958417 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-225f02d4-6e5d-4ab9-aa02-ef71f968c452 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:03.958897 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-225f02d4-6e5d-4ab9-aa02-ef71f968c452 None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:03.960103 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-225f02d4-6e5d-4ab9-aa02-ef71f968c452 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:03.960635 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-225f02d4-6e5d-4ab9-aa02-ef71f968c452 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:03.963134 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-225f02d4-6e5d-4ab9-aa02-ef71f968c452 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:03.987309 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 113/231] 10.222.0.90 () {56 vars in 1111 bytes} [Mon Aug 29 21:20:03 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 431 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:03.994506 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c87af5c-fd13-4208-bd06-e43313d90d3e None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:04.108031 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c87af5c-fd13-4208-bd06-e43313d90d3e None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:04.117756 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:04.144402 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e48e7bee-2953-472a-91d1-6fcdd21bb07d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:04.145623 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e48e7bee-2953-472a-91d1-6fcdd21bb07d None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:04.145955 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e48e7bee-2953-472a-91d1-6fcdd21bb07d None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:04.146229 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e48e7bee-2953-472a-91d1-6fcdd21bb07d None admin] PATH_INFO: `/v3/users/cce3ca7fa2c74842a2105931d0c0aa39` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:04.147404 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e48e7bee-2953-472a-91d1-6fcdd21bb07d None admin] RBAC: Authorizing `identity:get_user(user_id=cce3ca7fa2c74842a2105931d0c0aa39)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:04.148844 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e48e7bee-2953-472a-91d1-6fcdd21bb07d None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:04.150717 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 119/232] 10.222.0.90 () {56 vars in 1177 bytes} [Mon Aug 29 21:20:03 2022] GET /identity/v3/users/cce3ca7fa2c74842a2105931d0c0aa39 => generated 283 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:20:04.156264 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-225f02d4-6e5d-4ab9-aa02-ef71f968c452 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:04.172890 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:04.262107 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-225f02d4-6e5d-4ab9-aa02-ef71f968c452 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:04.298328 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fc76c7f-d0de-443e-8892-3f5dcfa15184 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:04.298968 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fc76c7f-d0de-443e-8892-3f5dcfa15184 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:04.299248 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fc76c7f-d0de-443e-8892-3f5dcfa15184 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:04.299526 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fc76c7f-d0de-443e-8892-3f5dcfa15184 None admin] PATH_INFO: `/v3/projects/f6d8dbb619904d8889100044bad1bef6` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:04.300685 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fc76c7f-d0de-443e-8892-3f5dcfa15184 None admin] RBAC: Authorizing `identity:get_project(project_id=f6d8dbb619904d8889100044bad1bef6)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:04.302778 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fc76c7f-d0de-443e-8892-3f5dcfa15184 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:04.304804 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 114/233] 10.222.0.90 () {56 vars in 1186 bytes} [Mon Aug 29 21:20:04 2022] GET /identity/v3/projects/f6d8dbb619904d8889100044bad1bef6 => generated 302 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:20:04.310564 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e48e7bee-2953-472a-91d1-6fcdd21bb07d None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:04.420776 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e48e7bee-2953-472a-91d1-6fcdd21bb07d None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:04.455688 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5c3cc08-ab1e-43b7-8072-2d2da01c27dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:04.456356 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5c3cc08-ab1e-43b7-8072-2d2da01c27dc None admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:04.456634 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5c3cc08-ab1e-43b7-8072-2d2da01c27dc None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:04.456911 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5c3cc08-ab1e-43b7-8072-2d2da01c27dc None admin] PATH_INFO: `/v3/projects/f6d8dbb619904d8889100044bad1bef6/users/cce3ca7fa2c74842a2105931d0c0aa39/roles/9745586777bf473bab7d887bf25bf236` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:04.459727 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5c3cc08-ab1e-43b7-8072-2d2da01c27dc None admin] RBAC: Authorizing `identity:create_grant(project_id=f6d8dbb619904d8889100044bad1bef6, user_id=cce3ca7fa2c74842a2105931d0c0aa39, role_id=9745586777bf473bab7d887bf25bf236)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:04.461244 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5c3cc08-ab1e-43b7-8072-2d2da01c27dc None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:04.478527 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 120/234] 10.222.0.90 () {58 vars in 1439 bytes} [Mon Aug 29 21:20:04 2022] PUT /identity/v3/projects/f6d8dbb619904d8889100044bad1bef6/users/cce3ca7fa2c74842a2105931d0c0aa39/roles/9745586777bf473bab7d887bf25bf236 => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 21:20:04.812781 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-164f48e0-cda4-4ac6-bce0-a5756a75ef96 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:04.813147 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-164f48e0-cda4-4ac6-bce0-a5756a75ef96 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:04.813691 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-164f48e0-cda4-4ac6-bce0-a5756a75ef96 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:04.815275 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 115/235] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:20:04 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:20:04.820871 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df7c0465-d8f4-4ff9-8b50-7d42a250bd8b None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:04.821161 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df7c0465-d8f4-4ff9-8b50-7d42a250bd8b None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:04.821432 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df7c0465-d8f4-4ff9-8b50-7d42a250bd8b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:04.847097 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-df7c0465-d8f4-4ff9-8b50-7d42a250bd8b None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:20:04.848873 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-df7c0465-d8f4-4ff9-8b50-7d42a250bd8b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:20:04.885430 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:04.940369 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 121/236] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:20:04 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 21:20:05.118892 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-977e49ca-d7e1-46d0-896a-245ee686b6d2 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:05.119407 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-977e49ca-d7e1-46d0-896a-245ee686b6d2 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:05.119746 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-977e49ca-d7e1-46d0-896a-245ee686b6d2 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:05.122510 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 116/237] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:20:05 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:20:05.130611 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddd09292-e7d2-4253-8049-909ecea7b739 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:05.130947 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddd09292-e7d2-4253-8049-909ecea7b739 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:05.131224 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddd09292-e7d2-4253-8049-909ecea7b739 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:05.158795 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-ddd09292-e7d2-4253-8049-909ecea7b739 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:20:05.160007 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-ddd09292-e7d2-4253-8049-909ecea7b739 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:20:05.244090 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 122/238] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:20:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 29 21:20:05.576305 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-977e49ca-d7e1-46d0-896a-245ee686b6d2 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:05.727515 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-088efc1f-80e0-4545-bef1-b387c9f2ec10 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:05.728019 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-088efc1f-80e0-4545-bef1-b387c9f2ec10 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:05.728092 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-088efc1f-80e0-4545-bef1-b387c9f2ec10 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:05.729485 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 123/239] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:20:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 29 21:20:05.737027 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a1528d8-e0b0-4552-84e2-7bdc0b767fcd None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:05.737314 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a1528d8-e0b0-4552-84e2-7bdc0b767fcd None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:05.737591 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a1528d8-e0b0-4552-84e2-7bdc0b767fcd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:05.754259 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:05.767541 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-3a1528d8-e0b0-4552-84e2-7bdc0b767fcd None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:20:05.768814 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-3a1528d8-e0b0-4552-84e2-7bdc0b767fcd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:20:05.787909 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-977e49ca-d7e1-46d0-896a-245ee686b6d2 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:05.841045 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 124/240] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:20:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 21:20:05.841546 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a1528d8-e0b0-4552-84e2-7bdc0b767fcd None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:05.861408 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-410e8eb1-5fe6-4f8d-b046-aa7bc855c8c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:05.862155 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-410e8eb1-5fe6-4f8d-b046-aa7bc855c8c7 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:05.862439 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-410e8eb1-5fe6-4f8d-b046-aa7bc855c8c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:05.862718 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-410e8eb1-5fe6-4f8d-b046-aa7bc855c8c7 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:05.874134 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:05.891753 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-410e8eb1-5fe6-4f8d-b046-aa7bc855c8c7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 21:20:05.904769 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-410e8eb1-5fe6-4f8d-b046-aa7bc855c8c7 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 29 21:20:05.904769 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:05.904769 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:05.904769 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:05.904769 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:05.904769 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:05.904769 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:05.904769 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:05.904769 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:05.904769 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:05.904769 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:05.904769 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:05.904769 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 21:20:05.904769 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 21:20:05.904769 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 21:20:05.904769 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 21:20:05.905686 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 21:20:05.905686 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 21:20:05.905686 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 21:20:05.905686 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 21:20:05.905686 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:05.905686 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:05.905686 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:05.905686 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:05.905686 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:05.905686 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:05.905686 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:05.905686 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:05.905686 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:05.905686 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:05.905686 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:05.905686 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:05.906535 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:05.906535 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:05.906535 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:05.906535 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:05.906535 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 21:20:05.906535 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 21:20:05.906535 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 21:20:05.906535 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 21:20:05.906535 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 21:20:05.906535 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 21:20:05.906535 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 29 21:20:05.906535 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:20:05.913535 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 117/241] 10.222.0.90 () {56 vars in 1099 bytes} [Mon Aug 29 21:20:05 2022] GET /identity/v3/roles/reader => generated 84 bytes in 338 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:20:05.919416 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-410e8eb1-5fe6-4f8d-b046-aa7bc855c8c7 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:05.993421 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a1528d8-e0b0-4552-84e2-7bdc0b767fcd None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:06.065369 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2965f20c-4f6a-4511-a144-ab2758df0591 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:06.066574 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2965f20c-4f6a-4511-a144-ab2758df0591 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:06.068718 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2965f20c-4f6a-4511-a144-ab2758df0591 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:06.069349 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2965f20c-4f6a-4511-a144-ab2758df0591 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:06.071514 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2965f20c-4f6a-4511-a144-ab2758df0591 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:06.073343 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2965f20c-4f6a-4511-a144-ab2758df0591 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:06.077831 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 125/242] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:20:05 2022] GET /identity/v3/domains/default => generated 199 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:06.082196 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2965f20c-4f6a-4511-a144-ab2758df0591 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:06.101000 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-410e8eb1-5fe6-4f8d-b046-aa7bc855c8c7 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:06.154181 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-491b4705-753c-4352-8493-d319e2cf103f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:06.155788 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-491b4705-753c-4352-8493-d319e2cf103f None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:06.156345 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-491b4705-753c-4352-8493-d319e2cf103f None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:06.156765 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-491b4705-753c-4352-8493-d319e2cf103f None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:06.157583 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-491b4705-753c-4352-8493-d319e2cf103f None admin] RBAC: Adding query filter params (name=reader) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:06.158321 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-491b4705-753c-4352-8493-d319e2cf103f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:06.160766 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-491b4705-753c-4352-8493-d319e2cf103f None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:06.171809 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 118/243] 10.222.0.90 () {56 vars in 1101 bytes} [Mon Aug 29 21:20:05 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 253 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:06.173148 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-491b4705-753c-4352-8493-d319e2cf103f None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:06.177437 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:06.227718 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:06.235967 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2965f20c-4f6a-4511-a144-ab2758df0591 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:06.272333 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c730c201-98ae-4ded-9aee-eeeb03af5bcc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:06.273302 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c730c201-98ae-4ded-9aee-eeeb03af5bcc None admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:06.273775 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c730c201-98ae-4ded-9aee-eeeb03af5bcc None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:06.274277 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c730c201-98ae-4ded-9aee-eeeb03af5bcc None admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:06.274994 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c730c201-98ae-4ded-9aee-eeeb03af5bcc None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:06.276613 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c730c201-98ae-4ded-9aee-eeeb03af5bcc None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:06.304089 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 126/244] 10.222.0.90 () {60 vars in 1132 bytes} [Mon Aug 29 21:20:06 2022] POST /identity/v3/users => generated 293 bytes in 222 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 21:20:06.311015 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c730c201-98ae-4ded-9aee-eeeb03af5bcc None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:06.329159 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-491b4705-753c-4352-8493-d319e2cf103f None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:06.372234 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad4d8d88-602f-45a9-a0c5-1db48e773f3f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:06.373267 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad4d8d88-602f-45a9-a0c5-1db48e773f3f None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:06.373721 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad4d8d88-602f-45a9-a0c5-1db48e773f3f None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:06.374171 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad4d8d88-602f-45a9-a0c5-1db48e773f3f None admin] PATH_INFO: `/v3/roles/member` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:06.388364 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad4d8d88-602f-45a9-a0c5-1db48e773f3f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 21:20:06.398775 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-ad4d8d88-602f-45a9-a0c5-1db48e773f3f None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 29 21:20:06.398775 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:06.398775 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:06.398775 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:06.398775 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:06.398775 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:06.398775 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:06.398775 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:06.398775 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:06.398775 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:06.398775 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:06.398775 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:06.398775 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 21:20:06.398775 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 21:20:06.398775 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 21:20:06.398775 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 21:20:06.399994 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 21:20:06.399994 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 21:20:06.399994 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 21:20:06.399994 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 21:20:06.399994 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:06.399994 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:06.399994 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:06.399994 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:06.399994 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:06.399994 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:06.399994 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:06.399994 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:06.399994 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:06.399994 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:06.399994 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:06.399994 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:06.400947 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:06.400947 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:06.400947 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:06.400947 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:06.400947 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 21:20:06.400947 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 21:20:06.400947 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 21:20:06.400947 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 21:20:06.400947 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 21:20:06.400947 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 21:20:06.400947 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 29 21:20:06.400947 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:20:06.403064 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 119/245] 10.222.0.90 () {56 vars in 1099 bytes} [Mon Aug 29 21:20:06 2022] GET /identity/v3/roles/member => generated 84 bytes in 231 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:20:06.408895 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad4d8d88-602f-45a9-a0c5-1db48e773f3f None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:06.512436 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c730c201-98ae-4ded-9aee-eeeb03af5bcc None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:06.558180 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:06.566459 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eafc2e7e-15ee-4705-982f-1dee118e0f62 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:06.567844 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eafc2e7e-15ee-4705-982f-1dee118e0f62 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:06.568097 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eafc2e7e-15ee-4705-982f-1dee118e0f62 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:06.568374 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eafc2e7e-15ee-4705-982f-1dee118e0f62 None admin] PATH_INFO: `/v3/users/0e9ebcc726ca470c99dd9b40ff8732d7` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:06.585065 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eafc2e7e-15ee-4705-982f-1dee118e0f62 None admin] RBAC: Authorizing `identity:get_user(user_id=0e9ebcc726ca470c99dd9b40ff8732d7)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:06.586868 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eafc2e7e-15ee-4705-982f-1dee118e0f62 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:06.588715 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 127/246] 10.222.0.90 () {56 vars in 1177 bytes} [Mon Aug 29 21:20:06 2022] GET /identity/v3/users/0e9ebcc726ca470c99dd9b40ff8732d7 => generated 297 bytes in 278 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:06.590209 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49d3d201-c0b9-4c69-8eab-3edf9b834160 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:06.591603 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49d3d201-c0b9-4c69-8eab-3edf9b834160 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:06.592083 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49d3d201-c0b9-4c69-8eab-3edf9b834160 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:06.593344 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 128/247] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:20:06 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:20:06.598589 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49d3d201-c0b9-4c69-8eab-3edf9b834160 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:06.611981 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:06.616559 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad4d8d88-602f-45a9-a0c5-1db48e773f3f None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:06.656259 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d57179bc-9736-4572-bd64-34af3ee260eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:06.656907 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d57179bc-9736-4572-bd64-34af3ee260eb None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:06.657148 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d57179bc-9736-4572-bd64-34af3ee260eb None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:06.657417 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d57179bc-9736-4572-bd64-34af3ee260eb None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:06.658884 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d57179bc-9736-4572-bd64-34af3ee260eb None admin] RBAC: Adding query filter params (name=member) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:06.659187 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d57179bc-9736-4572-bd64-34af3ee260eb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:06.661257 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d57179bc-9736-4572-bd64-34af3ee260eb None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:06.669848 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 120/248] 10.222.0.90 () {56 vars in 1101 bytes} [Mon Aug 29 21:20:06 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 261 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:06.670888 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e5a986e-186c-4841-981b-d39fe17632fc None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:06.671176 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e5a986e-186c-4841-981b-d39fe17632fc None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:06.671450 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e5a986e-186c-4841-981b-d39fe17632fc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:06.692858 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-8e5a986e-186c-4841-981b-d39fe17632fc None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:20:06.693647 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-8e5a986e-186c-4841-981b-d39fe17632fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:20:06.730949 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49d3d201-c0b9-4c69-8eab-3edf9b834160 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:06.775511 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c835c77-9ed7-488f-8fda-9b10908c26a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:06.776753 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c835c77-9ed7-488f-8fda-9b10908c26a8 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:06.777338 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c835c77-9ed7-488f-8fda-9b10908c26a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:06.777471 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c835c77-9ed7-488f-8fda-9b10908c26a8 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:06.783769 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 121/249] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:20:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 21:20:06.784462 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e5a986e-186c-4841-981b-d39fe17632fc None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:06.869409 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:06.873402 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c835c77-9ed7-488f-8fda-9b10908c26a8 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:06.876294 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c835c77-9ed7-488f-8fda-9b10908c26a8 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:06.890465 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:06.926597 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e5a986e-186c-4841-981b-d39fe17632fc None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:06.977508 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d176fc2a-f447-41f7-a3b2-e67c5b15601a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:06.978204 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d176fc2a-f447-41f7-a3b2-e67c5b15601a None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:06.978512 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d176fc2a-f447-41f7-a3b2-e67c5b15601a None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:06.978888 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d176fc2a-f447-41f7-a3b2-e67c5b15601a None admin] PATH_INFO: `/v3/users/1e04ff7645fa4bae8e6f0ddbf7d824ab` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:06.982485 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d176fc2a-f447-41f7-a3b2-e67c5b15601a None admin] RBAC: Authorizing `identity:get_user(user_id=1e04ff7645fa4bae8e6f0ddbf7d824ab)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:06.983962 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d176fc2a-f447-41f7-a3b2-e67c5b15601a None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:06.985674 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 122/250] 10.222.0.90 () {56 vars in 1177 bytes} [Mon Aug 29 21:20:06 2022] GET /identity/v3/users/1e04ff7645fa4bae8e6f0ddbf7d824ab => generated 297 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:06.992204 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d176fc2a-f447-41f7-a3b2-e67c5b15601a None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:07.068860 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 129/251] 10.222.0.90 () {58 vars in 1282 bytes} [Mon Aug 29 21:20:06 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 471 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 29 21:20:07.074258 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c835c77-9ed7-488f-8fda-9b10908c26a8 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:07.128127 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d176fc2a-f447-41f7-a3b2-e67c5b15601a None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:07.151964 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:07.164865 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76284f5e-6e40-4e45-97c3-39b6860e7c97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:07.165625 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76284f5e-6e40-4e45-97c3-39b6860e7c97 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:07.165894 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76284f5e-6e40-4e45-97c3-39b6860e7c97 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:07.166172 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76284f5e-6e40-4e45-97c3-39b6860e7c97 None admin] PATH_INFO: `/v3/projects/f6d8dbb619904d8889100044bad1bef6` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:07.167430 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76284f5e-6e40-4e45-97c3-39b6860e7c97 None admin] RBAC: Authorizing `identity:get_project(project_id=f6d8dbb619904d8889100044bad1bef6)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:07.168997 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76284f5e-6e40-4e45-97c3-39b6860e7c97 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:07.170956 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 123/252] 10.222.0.90 () {56 vars in 1186 bytes} [Mon Aug 29 21:20:06 2022] GET /identity/v3/projects/f6d8dbb619904d8889100044bad1bef6 => generated 302 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:20:07.172049 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d07f68ee-a878-47ea-a144-bbc5e0500efa None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:07.172320 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d07f68ee-a878-47ea-a144-bbc5e0500efa None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:07.172639 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d07f68ee-a878-47ea-a144-bbc5e0500efa None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:07.174133 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 124/253] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:20:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:20:07.180160 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bad13f49-bb23-435f-9489-dd2b938e228e None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:07.180405 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bad13f49-bb23-435f-9489-dd2b938e228e None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:07.180619 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bad13f49-bb23-435f-9489-dd2b938e228e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:07.205192 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-bad13f49-bb23-435f-9489-dd2b938e228e None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:20:07.207049 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-bad13f49-bb23-435f-9489-dd2b938e228e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:20:07.228493 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c835c77-9ed7-488f-8fda-9b10908c26a8 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:07.241754 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:07.279081 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef8dc32d-5ebd-4a14-8dea-2291e0203c1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:07.279730 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef8dc32d-5ebd-4a14-8dea-2291e0203c1b None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:07.280006 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef8dc32d-5ebd-4a14-8dea-2291e0203c1b None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:07.280276 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef8dc32d-5ebd-4a14-8dea-2291e0203c1b None admin] PATH_INFO: `/v3/projects/f6d8dbb619904d8889100044bad1bef6` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:07.282194 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef8dc32d-5ebd-4a14-8dea-2291e0203c1b None admin] RBAC: Authorizing `identity:get_project(project_id=f6d8dbb619904d8889100044bad1bef6)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:07.283642 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef8dc32d-5ebd-4a14-8dea-2291e0203c1b None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:07.285315 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 130/254] 10.222.0.90 () {56 vars in 1186 bytes} [Mon Aug 29 21:20:07 2022] GET /identity/v3/projects/f6d8dbb619904d8889100044bad1bef6 => generated 302 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:07.286602 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef8dc32d-5ebd-4a14-8dea-2291e0203c1b None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:07.303782 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 125/255] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:20:07 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 21:20:07.304651 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bad13f49-bb23-435f-9489-dd2b938e228e None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:07.414217 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef8dc32d-5ebd-4a14-8dea-2291e0203c1b None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:07.455615 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a56b166-f606-49e0-ab54-de2dcbb61723 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:07.456205 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a56b166-f606-49e0-ab54-de2dcbb61723 None admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:07.456484 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a56b166-f606-49e0-ab54-de2dcbb61723 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:07.456753 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a56b166-f606-49e0-ab54-de2dcbb61723 None admin] PATH_INFO: `/v3/projects/f6d8dbb619904d8889100044bad1bef6/users/1e04ff7645fa4bae8e6f0ddbf7d824ab/roles/02e242d5520740d4b77a5f4008c7eb57` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:07.459930 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a56b166-f606-49e0-ab54-de2dcbb61723 None admin] RBAC: Authorizing `identity:create_grant(project_id=f6d8dbb619904d8889100044bad1bef6, user_id=1e04ff7645fa4bae8e6f0ddbf7d824ab, role_id=02e242d5520740d4b77a5f4008c7eb57)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:07.461363 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a56b166-f606-49e0-ab54-de2dcbb61723 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:07.476820 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 131/256] 10.222.0.90 () {58 vars in 1439 bytes} [Mon Aug 29 21:20:07 2022] PUT /identity/v3/projects/f6d8dbb619904d8889100044bad1bef6/users/1e04ff7645fa4bae8e6f0ddbf7d824ab/roles/02e242d5520740d4b77a5f4008c7eb57 => generated 0 bytes in 191 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:20:07.477993 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a56b166-f606-49e0-ab54-de2dcbb61723 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:07.491517 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bad13f49-bb23-435f-9489-dd2b938e228e None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:07.546287 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:07.553256 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:07.597817 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-004efdc9-0cbc-4fb8-9138-6eca051cdde8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:07.598575 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-004efdc9-0cbc-4fb8-9138-6eca051cdde8 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:07.598855 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-004efdc9-0cbc-4fb8-9138-6eca051cdde8 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:07.599124 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-004efdc9-0cbc-4fb8-9138-6eca051cdde8 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:07.630799 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-004efdc9-0cbc-4fb8-9138-6eca051cdde8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 21:20:07.635751 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-004efdc9-0cbc-4fb8-9138-6eca051cdde8 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 21:20:07.635751 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:07.635751 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:07.635751 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:07.635751 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:07.635751 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:07.635751 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:07.635751 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:07.635751 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:07.635751 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:07.635751 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:07.635751 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:07.635751 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 21:20:07.635751 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 21:20:07.635751 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 21:20:07.635751 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 21:20:07.636700 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 21:20:07.636700 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 21:20:07.636700 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 21:20:07.636700 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 21:20:07.636700 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:07.636700 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:07.636700 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:07.636700 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:07.636700 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:07.636700 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:07.636700 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:07.636700 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:07.636700 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:07.636700 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:07.636700 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:07.636700 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:07.637509 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:07.637509 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:07.637509 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:07.637509 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:07.637509 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 21:20:07.637509 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 21:20:07.637509 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 21:20:07.637509 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 21:20:07.637509 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 21:20:07.637509 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 21:20:07.637509 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 21:20:07.637509 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:20:07.637509 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 126/257] 10.222.0.90 () {56 vars in 1096 bytes} [Mon Aug 29 21:20:07 2022] GET /identity/v3/roles/admin => generated 83 bytes in 333 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:20:07.642775 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-004efdc9-0cbc-4fb8-9138-6eca051cdde8 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:07.701017 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a56b166-f606-49e0-ab54-de2dcbb61723 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:07.745923 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83be859f-eb4e-4585-b8aa-c79ce5ee12ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:07.747361 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83be859f-eb4e-4585-b8aa-c79ce5ee12ba None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:07.747847 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83be859f-eb4e-4585-b8aa-c79ce5ee12ba None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:07.748361 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83be859f-eb4e-4585-b8aa-c79ce5ee12ba None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:07.749323 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83be859f-eb4e-4585-b8aa-c79ce5ee12ba None admin] RBAC: Adding query filter params (role.id=414a14ad48a345f89fd1045f5d8fd16c, scope.project.id=f6d8dbb619904d8889100044bad1bef6, user.id=0e9ebcc726ca470c99dd9b40ff8732d7) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:07.749979 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83be859f-eb4e-4585-b8aa-c79ce5ee12ba None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:07.751773 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83be859f-eb4e-4585-b8aa-c79ce5ee12ba None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:07.759311 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 132/258] 10.222.0.90 () {56 vars in 1374 bytes} [Mon Aug 29 21:20:07 2022] GET /identity/v3/role_assignments?user.id=0e9ebcc726ca470c99dd9b40ff8732d7&scope.project.id=f6d8dbb619904d8889100044bad1bef6&role.id=414a14ad48a345f89fd1045f5d8fd16c => generated 259 bytes in 282 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:07.760453 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83be859f-eb4e-4585-b8aa-c79ce5ee12ba None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:07.789512 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-004efdc9-0cbc-4fb8-9138-6eca051cdde8 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:07.878736 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df578d5e-c582-46df-b68f-8c316d23e915 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:07.879487 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df578d5e-c582-46df-b68f-8c316d23e915 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:07.879678 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df578d5e-c582-46df-b68f-8c316d23e915 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:07.879956 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df578d5e-c582-46df-b68f-8c316d23e915 None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:07.880641 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df578d5e-c582-46df-b68f-8c316d23e915 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:07.881013 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df578d5e-c582-46df-b68f-8c316d23e915 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:07.882757 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df578d5e-c582-46df-b68f-8c316d23e915 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:07.902375 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 127/259] 10.222.0.90 () {56 vars in 1099 bytes} [Mon Aug 29 21:20:07 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 260 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:20:07.911617 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df578d5e-c582-46df-b68f-8c316d23e915 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:07.934450 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:07.937532 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83be859f-eb4e-4585-b8aa-c79ce5ee12ba None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:07.974492 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3158613b-aa0c-4ad7-ae96-0a876c9327ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:07.975247 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3158613b-aa0c-4ad7-ae96-0a876c9327ad None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:07.975536 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3158613b-aa0c-4ad7-ae96-0a876c9327ad None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:07.975812 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3158613b-aa0c-4ad7-ae96-0a876c9327ad None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:07.981466 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:07.991178 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3158613b-aa0c-4ad7-ae96-0a876c9327ad None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 21:20:07.996964 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-3158613b-aa0c-4ad7-ae96-0a876c9327ad None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 29 21:20:07.996964 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:07.996964 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:07.996964 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:07.996964 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:07.996964 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:07.996964 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:07.996964 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:07.996964 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:07.996964 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:07.996964 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:07.996964 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:07.996964 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 21:20:07.996964 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 21:20:07.996964 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 21:20:07.996964 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 21:20:07.998218 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 21:20:07.998218 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 21:20:07.998218 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 21:20:07.998218 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 21:20:07.998218 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:07.998218 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:07.998218 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:07.998218 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:07.998218 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:07.998218 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:07.998218 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:07.998218 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:07.998218 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:07.998218 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:07.998218 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:07.998218 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:07.999268 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:07.999268 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:07.999268 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:07.999268 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:07.999268 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 21:20:07.999268 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 21:20:07.999268 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 21:20:07.999268 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 21:20:07.999268 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 21:20:07.999268 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 21:20:07.999268 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 29 21:20:07.999268 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:20:08.001098 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 133/260] 10.222.0.90 () {56 vars in 1114 bytes} [Mon Aug 29 21:20:07 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 241 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:20:08.007061 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3158613b-aa0c-4ad7-ae96-0a876c9327ad None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:08.075489 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df578d5e-c582-46df-b68f-8c316d23e915 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:08.155822 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5699485d-7330-427f-ae34-5e00aa7ce6ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:08.156893 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5699485d-7330-427f-ae34-5e00aa7ce6ec None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:08.158456 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5699485d-7330-427f-ae34-5e00aa7ce6ec None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:08.159461 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5699485d-7330-427f-ae34-5e00aa7ce6ec None admin] PATH_INFO: `/v3/users/cce3ca7fa2c74842a2105931d0c0aa39` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:08.166481 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5699485d-7330-427f-ae34-5e00aa7ce6ec None admin] RBAC: Authorizing `identity:get_user(user_id=cce3ca7fa2c74842a2105931d0c0aa39)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:08.168573 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5699485d-7330-427f-ae34-5e00aa7ce6ec None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:08.172916 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 128/261] 10.222.0.90 () {56 vars in 1177 bytes} [Mon Aug 29 21:20:07 2022] GET /identity/v3/users/cce3ca7fa2c74842a2105931d0c0aa39 => generated 283 bytes in 262 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:08.177205 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5699485d-7330-427f-ae34-5e00aa7ce6ec None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:08.179253 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3158613b-aa0c-4ad7-ae96-0a876c9327ad None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:08.268973 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc5d033a-e8d6-458c-a5b5-212054774b7b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:08.269579 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc5d033a-e8d6-458c-a5b5-212054774b7b None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:08.270498 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc5d033a-e8d6-458c-a5b5-212054774b7b None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:08.270783 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc5d033a-e8d6-458c-a5b5-212054774b7b None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:08.271476 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc5d033a-e8d6-458c-a5b5-212054774b7b None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:08.271877 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc5d033a-e8d6-458c-a5b5-212054774b7b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:08.273385 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc5d033a-e8d6-458c-a5b5-212054774b7b None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:08.292346 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 134/262] 10.222.0.90 () {56 vars in 1111 bytes} [Mon Aug 29 21:20:08 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 286 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:20:08.298430 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc5d033a-e8d6-458c-a5b5-212054774b7b None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:08.344627 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:08.393532 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5699485d-7330-427f-ae34-5e00aa7ce6ec None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:08.413327 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:08.482940 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9169cc6-f1fc-484a-9288-7b127cf2ee60 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:08.483565 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9169cc6-f1fc-484a-9288-7b127cf2ee60 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:08.483848 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9169cc6-f1fc-484a-9288-7b127cf2ee60 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:08.484123 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9169cc6-f1fc-484a-9288-7b127cf2ee60 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:08.524029 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc5d033a-e8d6-458c-a5b5-212054774b7b None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:08.587432 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6c79ec6-1c0f-4a41-b921-fa2d25641349 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:08.588084 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c79ec6-1c0f-4a41-b921-fa2d25641349 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:08.588366 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c79ec6-1c0f-4a41-b921-fa2d25641349 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:08.588641 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c79ec6-1c0f-4a41-b921-fa2d25641349 None admin] PATH_INFO: `/v3/users/cce3ca7fa2c74842a2105931d0c0aa39` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:08.592274 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6c79ec6-1c0f-4a41-b921-fa2d25641349 None admin] RBAC: Authorizing `identity:get_user(user_id=cce3ca7fa2c74842a2105931d0c0aa39)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:08.593869 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6c79ec6-1c0f-4a41-b921-fa2d25641349 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:08.595911 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 135/263] 10.222.0.90 () {56 vars in 1177 bytes} [Mon Aug 29 21:20:08 2022] GET /identity/v3/users/cce3ca7fa2c74842a2105931d0c0aa39 => generated 283 bytes in 298 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:20:08.602522 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6c79ec6-1c0f-4a41-b921-fa2d25641349 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:08.617315 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9169cc6-f1fc-484a-9288-7b127cf2ee60 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:08.619134 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9169cc6-f1fc-484a-9288-7b127cf2ee60 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:08.811439 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:08.811969 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 129/264] 10.222.0.90 () {58 vars in 1282 bytes} [Mon Aug 29 21:20:08 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 635 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 29 21:20:08.818306 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9169cc6-f1fc-484a-9288-7b127cf2ee60 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:08.837503 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6c79ec6-1c0f-4a41-b921-fa2d25641349 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:08.840675 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:08.899873 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c4673bd-1d0d-4e97-8205-3b3629084113 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:08.900896 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c4673bd-1d0d-4e97-8205-3b3629084113 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:08.901368 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c4673bd-1d0d-4e97-8205-3b3629084113 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:08.901853 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c4673bd-1d0d-4e97-8205-3b3629084113 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:09.022480 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9169cc6-f1fc-484a-9288-7b127cf2ee60 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:09.033871 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c4673bd-1d0d-4e97-8205-3b3629084113 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:09.036789 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c4673bd-1d0d-4e97-8205-3b3629084113 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:09.088677 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c211c0f-cf22-49c0-90a4-e79ab25ca328 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:09.089242 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c211c0f-cf22-49c0-90a4-e79ab25ca328 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:09.089524 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c211c0f-cf22-49c0-90a4-e79ab25ca328 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:09.089809 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c211c0f-cf22-49c0-90a4-e79ab25ca328 None admin] PATH_INFO: `/v3/projects/f6d8dbb619904d8889100044bad1bef6` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:09.092213 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c211c0f-cf22-49c0-90a4-e79ab25ca328 None admin] RBAC: Authorizing `identity:get_project(project_id=f6d8dbb619904d8889100044bad1bef6)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:09.093683 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c211c0f-cf22-49c0-90a4-e79ab25ca328 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:09.095391 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 130/265] 10.222.0.90 () {56 vars in 1186 bytes} [Mon Aug 29 21:20:08 2022] GET /identity/v3/projects/f6d8dbb619904d8889100044bad1bef6 => generated 302 bytes in 278 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:09.106920 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c211c0f-cf22-49c0-90a4-e79ab25ca328 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:09.222601 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:09.245245 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 136/266] 10.222.0.90 () {58 vars in 1282 bytes} [Mon Aug 29 21:20:08 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 643 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Aug 29 21:20:09.246887 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3de36675-414d-4159-bf1a-09d0ac5f450d None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:09.247172 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3de36675-414d-4159-bf1a-09d0ac5f450d None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:09.247444 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3de36675-414d-4159-bf1a-09d0ac5f450d None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:09.248814 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 137/267] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:20:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:20:09.254834 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3de36675-414d-4159-bf1a-09d0ac5f450d None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:09.259089 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:09.306178 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c211c0f-cf22-49c0-90a4-e79ab25ca328 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:09.374613 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86a83340-3b8a-4ff3-bca8-8e08b1d38a00 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:09.375605 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a83340-3b8a-4ff3-bca8-8e08b1d38a00 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:09.376106 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a83340-3b8a-4ff3-bca8-8e08b1d38a00 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:09.376578 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a83340-3b8a-4ff3-bca8-8e08b1d38a00 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:09.377536 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86a83340-3b8a-4ff3-bca8-8e08b1d38a00 None admin] RBAC: Adding query filter params (role.id=a6e562cdc96c4be691d638990f8403fd, scope.project.id=f6d8dbb619904d8889100044bad1bef6, user.id=cce3ca7fa2c74842a2105931d0c0aa39) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:09.378071 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86a83340-3b8a-4ff3-bca8-8e08b1d38a00 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:09.380663 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86a83340-3b8a-4ff3-bca8-8e08b1d38a00 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:09.387221 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 131/268] 10.222.0.90 () {56 vars in 1374 bytes} [Mon Aug 29 21:20:09 2022] GET /identity/v3/role_assignments?user.id=cce3ca7fa2c74842a2105931d0c0aa39&scope.project.id=f6d8dbb619904d8889100044bad1bef6&role.id=a6e562cdc96c4be691d638990f8403fd => generated 608 bytes in 284 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:20:09.393818 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e0b358-9fc2-4f40-a95f-86f38cd9d64f None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:09.395773 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e0b358-9fc2-4f40-a95f-86f38cd9d64f None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:09.396729 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e0b358-9fc2-4f40-a95f-86f38cd9d64f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:09.442496 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-e0e0b358-9fc2-4f40-a95f-86f38cd9d64f None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:20:09.444098 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-e0e0b358-9fc2-4f40-a95f-86f38cd9d64f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:20:09.466562 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3de36675-414d-4159-bf1a-09d0ac5f450d None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:09.526145 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66617cb3-428e-4702-bd05-46e8ac181f85 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:09.526892 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66617cb3-428e-4702-bd05-46e8ac181f85 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:09.527178 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66617cb3-428e-4702-bd05-46e8ac181f85 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:09.527448 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66617cb3-428e-4702-bd05-46e8ac181f85 None admin] PATH_INFO: `/v3/projects/f6d8dbb619904d8889100044bad1bef6` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:09.528575 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66617cb3-428e-4702-bd05-46e8ac181f85 None admin] RBAC: Authorizing `identity:get_project(project_id=f6d8dbb619904d8889100044bad1bef6)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:09.535114 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66617cb3-428e-4702-bd05-46e8ac181f85 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:09.537968 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 138/269] 10.222.0.90 () {56 vars in 1186 bytes} [Mon Aug 29 21:20:09 2022] GET /identity/v3/projects/f6d8dbb619904d8889100044bad1bef6 => generated 302 bytes in 283 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:09.544287 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66617cb3-428e-4702-bd05-46e8ac181f85 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:09.600827 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 132/270] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:20:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 213 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 21:20:09.641853 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:09.675283 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d33b9c-284a-4a62-b3ab-9885ebfb6af0 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:09.675584 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d33b9c-284a-4a62-b3ab-9885ebfb6af0 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:09.675834 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d33b9c-284a-4a62-b3ab-9885ebfb6af0 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:09.681951 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 133/271] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:20:09 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 29 21:20:09.683149 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c68fd2a-3277-40ae-abe6-5970ef8f3b25 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:09.683437 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c68fd2a-3277-40ae-abe6-5970ef8f3b25 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:09.683712 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c68fd2a-3277-40ae-abe6-5970ef8f3b25 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:09.686361 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66617cb3-428e-4702-bd05-46e8ac181f85 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:09.706007 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-1c68fd2a-3277-40ae-abe6-5970ef8f3b25 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:20:09.707199 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-1c68fd2a-3277-40ae-abe6-5970ef8f3b25 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:20:09.722624 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a356ecd-3d09-4f6e-a70d-f7b224733598 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:09.723931 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a356ecd-3d09-4f6e-a70d-f7b224733598 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:09.724195 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a356ecd-3d09-4f6e-a70d-f7b224733598 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:09.724530 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a356ecd-3d09-4f6e-a70d-f7b224733598 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:09.725211 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a356ecd-3d09-4f6e-a70d-f7b224733598 None admin] RBAC: Adding query filter params (role.id=9745586777bf473bab7d887bf25bf236, scope.project.id=f6d8dbb619904d8889100044bad1bef6, user.id=cce3ca7fa2c74842a2105931d0c0aa39) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:09.725516 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a356ecd-3d09-4f6e-a70d-f7b224733598 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:09.727101 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a356ecd-3d09-4f6e-a70d-f7b224733598 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:09.733549 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 139/272] 10.222.0.90 () {56 vars in 1374 bytes} [Mon Aug 29 21:20:09 2022] GET /identity/v3/role_assignments?user.id=cce3ca7fa2c74842a2105931d0c0aa39&scope.project.id=f6d8dbb619904d8889100044bad1bef6&role.id=9745586777bf473bab7d887bf25bf236 => generated 608 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:20:09.813519 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 134/273] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:20:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 29 21:20:09.982762 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a356ecd-3d09-4f6e-a70d-f7b224733598 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:10.089225 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a356ecd-3d09-4f6e-a70d-f7b224733598 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:10.114908 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:10.125255 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1d8bd9e-bcff-4833-a403-0cf847db50bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:10.125982 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1d8bd9e-bcff-4833-a403-0cf847db50bc None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:10.126261 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1d8bd9e-bcff-4833-a403-0cf847db50bc None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:10.126531 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1d8bd9e-bcff-4833-a403-0cf847db50bc None admin] PATH_INFO: `/v3/roles/member` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:10.137828 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1d8bd9e-bcff-4833-a403-0cf847db50bc None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 21:20:10.143130 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-f1d8bd9e-bcff-4833-a403-0cf847db50bc None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 29 21:20:10.143130 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:10.143130 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:10.143130 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:10.143130 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:10.143130 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:10.143130 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:10.143130 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:10.143130 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:10.143130 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:10.143130 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:10.143130 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:10.143130 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 21:20:10.143130 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 21:20:10.143130 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 21:20:10.143130 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 21:20:10.144032 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 21:20:10.144032 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 21:20:10.144032 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 21:20:10.144032 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 21:20:10.144032 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:10.144032 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:10.144032 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:10.144032 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:10.144032 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:10.144032 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:10.144032 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:10.144032 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:10.144032 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:10.144032 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:10.144032 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:10.144032 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:10.144945 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:10.144945 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:10.144945 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:10.144945 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:10.144945 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 21:20:10.144945 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 21:20:10.144945 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 21:20:10.144945 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 21:20:10.144945 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 21:20:10.144945 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 21:20:10.144945 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 29 21:20:10.144945 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:20:10.144945 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 140/274] 10.222.0.90 () {56 vars in 1099 bytes} [Mon Aug 29 21:20:09 2022] GET /identity/v3/roles/member => generated 84 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:20:10.150929 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c68fd2a-3277-40ae-abe6-5970ef8f3b25 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:10.153538 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:10.175920 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1d8bd9e-bcff-4833-a403-0cf847db50bc None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:10.279745 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c68fd2a-3277-40ae-abe6-5970ef8f3b25 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:10.310241 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1d8bd9e-bcff-4833-a403-0cf847db50bc None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:10.331517 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0ed820a-b925-4e8f-abd8-f4f71ec7995a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:10.332533 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0ed820a-b925-4e8f-abd8-f4f71ec7995a None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:10.333021 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0ed820a-b925-4e8f-abd8-f4f71ec7995a None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:10.333519 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0ed820a-b925-4e8f-abd8-f4f71ec7995a None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:10.334414 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0ed820a-b925-4e8f-abd8-f4f71ec7995a None admin] RBAC: Adding query filter params (name=member) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:10.334866 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0ed820a-b925-4e8f-abd8-f4f71ec7995a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:10.336411 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0ed820a-b925-4e8f-abd8-f4f71ec7995a None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:10.348559 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 135/275] 10.222.0.90 () {56 vars in 1101 bytes} [Mon Aug 29 21:20:10 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:20:10.353409 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be7277be-b5d7-46f0-ada6-56049ec5d19c None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:10.353409 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be7277be-b5d7-46f0-ada6-56049ec5d19c None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:10.353409 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be7277be-b5d7-46f0-ada6-56049ec5d19c None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:10.353409 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 136/276] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:20:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:20:10.353697 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e22231ff-b6ac-4a05-8e22-173b4760e22c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:10.353697 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e22231ff-b6ac-4a05-8e22-173b4760e22c None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:10.353886 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e22231ff-b6ac-4a05-8e22-173b4760e22c None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:10.354223 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e22231ff-b6ac-4a05-8e22-173b4760e22c None admin] PATH_INFO: `/v3/domains/default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:10.355209 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e22231ff-b6ac-4a05-8e22-173b4760e22c None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:10.356822 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e22231ff-b6ac-4a05-8e22-173b4760e22c None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:10.357033 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be7277be-b5d7-46f0-ada6-56049ec5d19c None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:10.358570 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 141/277] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:20:10 2022] GET /identity/v3/domains/default => generated 199 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:20:10.363278 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d33d75-e738-49b6-9341-d611cd6cc3da None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:10.364165 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d33d75-e738-49b6-9341-d611cd6cc3da None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:10.364505 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d33d75-e738-49b6-9341-d611cd6cc3da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:10.383256 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-a9d33d75-e738-49b6-9341-d611cd6cc3da None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:20:10.384465 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-a9d33d75-e738-49b6-9341-d611cd6cc3da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:20:10.426043 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:10.437145 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:10.461273 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 142/278] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:20:10 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 21:20:10.462762 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9d33d75-e738-49b6-9341-d611cd6cc3da None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:10.482765 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be7277be-b5d7-46f0-ada6-56049ec5d19c None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:10.521597 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-792993ad-5eed-4358-927e-34a0472b5672 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:10.522493 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-792993ad-5eed-4358-927e-34a0472b5672 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:10.522775 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-792993ad-5eed-4358-927e-34a0472b5672 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:10.523041 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-792993ad-5eed-4358-927e-34a0472b5672 None admin] PATH_INFO: `/v3/users/ff1749b05c364ca199eea9870871e691` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:10.536967 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-792993ad-5eed-4358-927e-34a0472b5672 None admin] RBAC: Authorizing `identity:get_user(user_id=ff1749b05c364ca199eea9870871e691)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:10.538447 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-792993ad-5eed-4358-927e-34a0472b5672 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:10.540237 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 137/279] 10.222.0.90 () {56 vars in 1177 bytes} [Mon Aug 29 21:20:10 2022] GET /identity/v3/users/ff1749b05c364ca199eea9870871e691 => generated 293 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:10.545953 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fdf5f9b-26c9-4b03-a2e9-1de8c676c2e4 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:10.546236 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fdf5f9b-26c9-4b03-a2e9-1de8c676c2e4 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:10.546504 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fdf5f9b-26c9-4b03-a2e9-1de8c676c2e4 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:10.547737 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 138/280] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:20:10 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:20:10.549904 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fdf5f9b-26c9-4b03-a2e9-1de8c676c2e4 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:10.580395 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9d33d75-e738-49b6-9341-d611cd6cc3da None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:10.617381 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3074bae1-b30e-4e3d-9316-03843b37437c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:10.618029 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3074bae1-b30e-4e3d-9316-03843b37437c None admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:10.618311 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3074bae1-b30e-4e3d-9316-03843b37437c None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:10.618584 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3074bae1-b30e-4e3d-9316-03843b37437c None admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:10.619235 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3074bae1-b30e-4e3d-9316-03843b37437c None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:10.620667 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3074bae1-b30e-4e3d-9316-03843b37437c None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:10.642061 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 143/281] 10.222.0.90 () {60 vars in 1132 bytes} [Mon Aug 29 21:20:10 2022] POST /identity/v3/users => generated 293 bytes in 180 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:20:10.643388 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8552261d-04a8-4b86-987f-eb9ddd486138 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:10.643666 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8552261d-04a8-4b86-987f-eb9ddd486138 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:10.643940 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8552261d-04a8-4b86-987f-eb9ddd486138 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:10.664574 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-8552261d-04a8-4b86-987f-eb9ddd486138 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:20:10.665833 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-8552261d-04a8-4b86-987f-eb9ddd486138 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:20:10.666738 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fdf5f9b-26c9-4b03-a2e9-1de8c676c2e4 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:10.704016 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:10.720485 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:10.726780 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9c6f7ef-46cf-4667-bd6a-aec0caaf7117 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:10.727502 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9c6f7ef-46cf-4667-bd6a-aec0caaf7117 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:10.727702 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9c6f7ef-46cf-4667-bd6a-aec0caaf7117 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:10.728056 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9c6f7ef-46cf-4667-bd6a-aec0caaf7117 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:10.728666 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9c6f7ef-46cf-4667-bd6a-aec0caaf7117 None admin] RBAC: Adding query filter params (role.id=02e242d5520740d4b77a5f4008c7eb57, user.id=ff1749b05c364ca199eea9870871e691, scope.system=all) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:10.728974 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9c6f7ef-46cf-4667-bd6a-aec0caaf7117 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:10.730460 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9c6f7ef-46cf-4667-bd6a-aec0caaf7117 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:10.745827 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 139/282] 10.222.0.90 () {56 vars in 1308 bytes} [Mon Aug 29 21:20:10 2022] GET /identity/v3/role_assignments?user.id=ff1749b05c364ca199eea9870871e691&scope.system=all&role.id=02e242d5520740d4b77a5f4008c7eb57 => generated 226 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:10.745827 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9c6f7ef-46cf-4667-bd6a-aec0caaf7117 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:10.792701 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 144/283] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:20:10 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 150 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 21:20:10.889633 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9c6f7ef-46cf-4667-bd6a-aec0caaf7117 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:10.927414 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9eb8330-d7bb-4155-8c9f-13f33b99d829 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:10.928059 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9eb8330-d7bb-4155-8c9f-13f33b99d829 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:10.928361 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9eb8330-d7bb-4155-8c9f-13f33b99d829 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:10.928633 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9eb8330-d7bb-4155-8c9f-13f33b99d829 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:10.939758 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9eb8330-d7bb-4155-8c9f-13f33b99d829 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 21:20:10.946500 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-f9eb8330-d7bb-4155-8c9f-13f33b99d829 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 29 21:20:10.946500 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:10.946500 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:10.946500 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:10.946500 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:10.946500 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:10.946500 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:10.946500 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:10.946500 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:10.946500 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:10.946500 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:10.946500 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:10.946500 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 21:20:10.946500 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 21:20:10.946500 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 21:20:10.946500 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 21:20:10.947449 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 21:20:10.947449 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 21:20:10.947449 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 21:20:10.947449 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 21:20:10.947449 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:10.947449 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:10.947449 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:10.947449 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:10.947449 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:10.947449 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:10.947449 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:10.947449 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:10.947449 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:10.947449 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:10.947449 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:10.947449 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:10.948281 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:10.948281 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:10.948281 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:10.948281 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:10.948281 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 21:20:10.948281 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 21:20:10.948281 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 21:20:10.948281 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 21:20:10.948281 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 21:20:10.948281 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 21:20:10.948281 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 29 21:20:10.948281 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:20:10.948281 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 140/284] 10.222.0.90 () {56 vars in 1099 bytes} [Mon Aug 29 21:20:10 2022] GET /identity/v3/roles/member => generated 84 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 29 21:20:10.955583 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8552261d-04a8-4b86-987f-eb9ddd486138 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:11.069598 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8552261d-04a8-4b86-987f-eb9ddd486138 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:11.078939 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9eb8330-d7bb-4155-8c9f-13f33b99d829 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:11.113515 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ead69f41-f80c-4f6e-a2aa-41f2d7a57414 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:11.114444 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ead69f41-f80c-4f6e-a2aa-41f2d7a57414 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:11.114726 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ead69f41-f80c-4f6e-a2aa-41f2d7a57414 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:11.114993 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ead69f41-f80c-4f6e-a2aa-41f2d7a57414 None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:11.115673 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ead69f41-f80c-4f6e-a2aa-41f2d7a57414 None admin] RBAC: Adding query filter params (name=member) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:11.116093 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ead69f41-f80c-4f6e-a2aa-41f2d7a57414 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:11.117597 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ead69f41-f80c-4f6e-a2aa-41f2d7a57414 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:11.126977 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 145/285] 10.222.0.90 () {56 vars in 1101 bytes} [Mon Aug 29 21:20:10 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:11.138385 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ead69f41-f80c-4f6e-a2aa-41f2d7a57414 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:11.139966 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:11.172504 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:11.199116 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9eb8330-d7bb-4155-8c9f-13f33b99d829 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:11.235703 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-167788a4-710f-452e-abdf-5d162b8ab1b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:11.236355 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-167788a4-710f-452e-abdf-5d162b8ab1b7 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:11.236640 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-167788a4-710f-452e-abdf-5d162b8ab1b7 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:11.236909 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-167788a4-710f-452e-abdf-5d162b8ab1b7 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:11.248512 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-167788a4-710f-452e-abdf-5d162b8ab1b7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 21:20:11.253712 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-167788a4-710f-452e-abdf-5d162b8ab1b7 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 29 21:20:11.253712 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:11.253712 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:11.253712 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:11.253712 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:11.253712 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:11.253712 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:11.253712 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:11.253712 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:11.253712 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:11.253712 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:11.253712 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:11.253712 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 21:20:11.253712 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 21:20:11.253712 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 21:20:11.253712 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 21:20:11.255259 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 21:20:11.255259 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 21:20:11.255259 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 21:20:11.255259 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 21:20:11.255259 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:11.255259 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:11.255259 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:11.255259 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:11.255259 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:11.255259 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:11.255259 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:11.255259 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:11.255259 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:11.255259 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:11.255259 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:11.255259 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:11.256537 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:11.256537 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:11.256537 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:11.256537 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:11.256537 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 21:20:11.256537 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 21:20:11.256537 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 21:20:11.256537 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 21:20:11.256537 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 21:20:11.256537 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 21:20:11.256537 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 29 21:20:11.256537 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:20:11.256537 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 141/286] 10.222.0.90 () {56 vars in 1099 bytes} [Mon Aug 29 21:20:11 2022] GET /identity/v3/roles/reader => generated 84 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 29 21:20:11.261611 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-167788a4-710f-452e-abdf-5d162b8ab1b7 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:11.318998 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ead69f41-f80c-4f6e-a2aa-41f2d7a57414 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:11.369763 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-167788a4-710f-452e-abdf-5d162b8ab1b7 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:11.377371 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:11.407143 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-985ab1cd-bb5e-43c5-b361-0be89b39c078 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:11.407771 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-985ab1cd-bb5e-43c5-b361-0be89b39c078 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:11.408056 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-985ab1cd-bb5e-43c5-b361-0be89b39c078 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:11.408335 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-985ab1cd-bb5e-43c5-b361-0be89b39c078 None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:11.409016 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-985ab1cd-bb5e-43c5-b361-0be89b39c078 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:11.409322 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-985ab1cd-bb5e-43c5-b361-0be89b39c078 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:11.411143 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-985ab1cd-bb5e-43c5-b361-0be89b39c078 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:11.411349 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-263ebff4-94b3-4a51-b65d-5d59f6eba48e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:11.411977 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-263ebff4-94b3-4a51-b65d-5d59f6eba48e None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:11.412261 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-263ebff4-94b3-4a51-b65d-5d59f6eba48e None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:11.412534 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-263ebff4-94b3-4a51-b65d-5d59f6eba48e None admin] PATH_INFO: `/v3/users/1e04ff7645fa4bae8e6f0ddbf7d824ab` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:11.420353 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-263ebff4-94b3-4a51-b65d-5d59f6eba48e None admin] RBAC: Authorizing `identity:get_user(user_id=1e04ff7645fa4bae8e6f0ddbf7d824ab)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:11.420353 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-263ebff4-94b3-4a51-b65d-5d59f6eba48e None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:11.420353 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 146/287] 10.222.0.90 () {56 vars in 1177 bytes} [Mon Aug 29 21:20:11 2022] GET /identity/v3/users/1e04ff7645fa4bae8e6f0ddbf7d824ab => generated 297 bytes in 282 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:11.420631 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 142/288] 10.222.0.90 () {56 vars in 1101 bytes} [Mon Aug 29 21:20:11 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:20:11.425867 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-263ebff4-94b3-4a51-b65d-5d59f6eba48e None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:11.432112 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-985ab1cd-bb5e-43c5-b361-0be89b39c078 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:11.540835 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-985ab1cd-bb5e-43c5-b361-0be89b39c078 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:11.541597 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:11.557541 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-263ebff4-94b3-4a51-b65d-5d59f6eba48e None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:11.579119 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6a13694-05c3-408e-8678-b7a7344c3f1d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:11.579815 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6a13694-05c3-408e-8678-b7a7344c3f1d None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:11.580177 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6a13694-05c3-408e-8678-b7a7344c3f1d None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:11.580460 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6a13694-05c3-408e-8678-b7a7344c3f1d None admin] PATH_INFO: `/v3/users/0e9ebcc726ca470c99dd9b40ff8732d7` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:11.581726 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6a13694-05c3-408e-8678-b7a7344c3f1d None admin] RBAC: Authorizing `identity:get_user(user_id=0e9ebcc726ca470c99dd9b40ff8732d7)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:11.585969 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6a13694-05c3-408e-8678-b7a7344c3f1d None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:11.585969 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 143/289] 10.222.0.90 () {56 vars in 1177 bytes} [Mon Aug 29 21:20:11 2022] GET /identity/v3/users/0e9ebcc726ca470c99dd9b40ff8732d7 => generated 297 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:20:11.591649 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6a13694-05c3-408e-8678-b7a7344c3f1d None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:11.627918 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-139e67d9-87ba-4cee-9ae2-9faa5b639a87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:11.631228 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-139e67d9-87ba-4cee-9ae2-9faa5b639a87 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:11.631518 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-139e67d9-87ba-4cee-9ae2-9faa5b639a87 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:11.631794 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-139e67d9-87ba-4cee-9ae2-9faa5b639a87 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:11.662164 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:11.752915 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-139e67d9-87ba-4cee-9ae2-9faa5b639a87 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:11.754984 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-139e67d9-87ba-4cee-9ae2-9faa5b639a87 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:11.764999 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6a13694-05c3-408e-8678-b7a7344c3f1d None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:11.828724 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c15794c7-664c-473f-84c5-ea9986e99acb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:11.830076 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c15794c7-664c-473f-84c5-ea9986e99acb None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:11.830569 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c15794c7-664c-473f-84c5-ea9986e99acb None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:11.831044 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c15794c7-664c-473f-84c5-ea9986e99acb None admin] PATH_INFO: `/v3/projects/f6d8dbb619904d8889100044bad1bef6` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:11.837820 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c15794c7-664c-473f-84c5-ea9986e99acb None admin] RBAC: Authorizing `identity:get_project(project_id=f6d8dbb619904d8889100044bad1bef6)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:11.837820 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c15794c7-664c-473f-84c5-ea9986e99acb None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:11.837820 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 144/290] 10.222.0.90 () {56 vars in 1186 bytes} [Mon Aug 29 21:20:11 2022] GET /identity/v3/projects/f6d8dbb619904d8889100044bad1bef6 => generated 302 bytes in 247 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:20:11.843230 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c15794c7-664c-473f-84c5-ea9986e99acb None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:11.918269 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:11.930119 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 147/291] 10.222.0.90 () {58 vars in 1282 bytes} [Mon Aug 29 21:20:11 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 505 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Aug 29 21:20:11.936324 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-139e67d9-87ba-4cee-9ae2-9faa5b639a87 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:12.047014 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c15794c7-664c-473f-84c5-ea9986e99acb None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:12.049835 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:12.084982 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-139e67d9-87ba-4cee-9ae2-9faa5b639a87 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:12.093474 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-141331e4-c0f6-485a-b763-ccd5120366d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:12.094191 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-141331e4-c0f6-485a-b763-ccd5120366d1 None admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:12.094477 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-141331e4-c0f6-485a-b763-ccd5120366d1 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:12.094751 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-141331e4-c0f6-485a-b763-ccd5120366d1 None admin] PATH_INFO: `/v3/projects/f6d8dbb619904d8889100044bad1bef6/users/0e9ebcc726ca470c99dd9b40ff8732d7/roles/414a14ad48a345f89fd1045f5d8fd16c` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:12.106598 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-141331e4-c0f6-485a-b763-ccd5120366d1 None admin] RBAC: Authorizing `identity:create_grant(project_id=f6d8dbb619904d8889100044bad1bef6, user_id=0e9ebcc726ca470c99dd9b40ff8732d7, role_id=414a14ad48a345f89fd1045f5d8fd16c)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:12.108068 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-141331e4-c0f6-485a-b763-ccd5120366d1 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:12.126675 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 145/292] 10.222.0.90 () {58 vars in 1439 bytes} [Mon Aug 29 21:20:11 2022] PUT /identity/v3/projects/f6d8dbb619904d8889100044bad1bef6/users/0e9ebcc726ca470c99dd9b40ff8732d7/roles/414a14ad48a345f89fd1045f5d8fd16c => generated 0 bytes in 284 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 21:20:12.134591 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e12608c-ead6-42a7-a1c8-9d86b3c95a49 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:12.135318 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e12608c-ead6-42a7-a1c8-9d86b3c95a49 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:12.135607 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e12608c-ead6-42a7-a1c8-9d86b3c95a49 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:12.135877 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e12608c-ead6-42a7-a1c8-9d86b3c95a49 None admin] PATH_INFO: `/v3/projects/f6d8dbb619904d8889100044bad1bef6` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:12.137206 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e12608c-ead6-42a7-a1c8-9d86b3c95a49 None admin] RBAC: Authorizing `identity:get_project(project_id=f6d8dbb619904d8889100044bad1bef6)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:12.138708 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e12608c-ead6-42a7-a1c8-9d86b3c95a49 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:12.140679 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 148/293] 10.222.0.90 () {56 vars in 1186 bytes} [Mon Aug 29 21:20:11 2022] GET /identity/v3/projects/f6d8dbb619904d8889100044bad1bef6 => generated 302 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:12.147018 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-141331e4-c0f6-485a-b763-ccd5120366d1 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:12.274195 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-141331e4-c0f6-485a-b763-ccd5120366d1 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:12.310627 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beef7e6e-6732-4ee9-be43-21f5b5e98af6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:12.311244 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beef7e6e-6732-4ee9-be43-21f5b5e98af6 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:12.311524 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beef7e6e-6732-4ee9-be43-21f5b5e98af6 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:12.311788 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beef7e6e-6732-4ee9-be43-21f5b5e98af6 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:12.312701 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beef7e6e-6732-4ee9-be43-21f5b5e98af6 None admin] RBAC: Adding query filter params (role.id=02e242d5520740d4b77a5f4008c7eb57, scope.project.id=f6d8dbb619904d8889100044bad1bef6, user.id=1e04ff7645fa4bae8e6f0ddbf7d824ab) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:12.313016 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beef7e6e-6732-4ee9-be43-21f5b5e98af6 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:12.314496 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beef7e6e-6732-4ee9-be43-21f5b5e98af6 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:12.321906 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 146/294] 10.222.0.90 () {56 vars in 1374 bytes} [Mon Aug 29 21:20:12 2022] GET /identity/v3/role_assignments?user.id=1e04ff7645fa4bae8e6f0ddbf7d824ab&scope.project.id=f6d8dbb619904d8889100044bad1bef6&role.id=02e242d5520740d4b77a5f4008c7eb57 => generated 608 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:13.006704 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-812573e2-73dd-457a-ac5d-5576303d56f4 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:13.007033 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-812573e2-73dd-457a-ac5d-5576303d56f4 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:13.007295 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-812573e2-73dd-457a-ac5d-5576303d56f4 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:13.008911 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 149/295] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:20:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:20:13.015610 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82b959ac-b42f-467f-a437-85d9c2ca50b7 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:13.016068 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82b959ac-b42f-467f-a437-85d9c2ca50b7 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:13.016349 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82b959ac-b42f-467f-a437-85d9c2ca50b7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:13.036535 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-82b959ac-b42f-467f-a437-85d9c2ca50b7 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:20:13.037741 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-82b959ac-b42f-467f-a437-85d9c2ca50b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:20:13.061711 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:13.109444 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 147/296] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:20:13 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 21:20:13.152891 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0918c0f1-125a-4fcc-b8d4-5b100d1149cf None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:13.153428 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0918c0f1-125a-4fcc-b8d4-5b100d1149cf None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:13.153428 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0918c0f1-125a-4fcc-b8d4-5b100d1149cf None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:13.154780 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 150/297] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:20:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:20:13.159975 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eadacb3-f6a0-4781-a6aa-a96d261dd669 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:13.160289 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eadacb3-f6a0-4781-a6aa-a96d261dd669 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:13.160777 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eadacb3-f6a0-4781-a6aa-a96d261dd669 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:13.165080 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e74a5304-1d11-40a8-a564-0decc8c489ca None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:13.165523 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e74a5304-1d11-40a8-a564-0decc8c489ca None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:13.166054 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e74a5304-1d11-40a8-a564-0decc8c489ca None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:13.167679 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 151/298] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:20:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:20:13.172630 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e0b0b3-181b-41b7-adf5-0b4a48332c3f None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:13.173159 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e0b0b3-181b-41b7-adf5-0b4a48332c3f None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:13.173626 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e0b0b3-181b-41b7-adf5-0b4a48332c3f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:13.180056 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-1eadacb3-f6a0-4781-a6aa-a96d261dd669 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:20:13.181497 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-1eadacb3-f6a0-4781-a6aa-a96d261dd669 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:20:13.193058 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-d0e0b0b3-181b-41b7-adf5-0b4a48332c3f None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:20:13.194569 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-d0e0b0b3-181b-41b7-adf5-0b4a48332c3f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:20:13.261042 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 148/299] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:20:13 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 21:20:13.261234 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:13.268186 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 152/300] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:20:13 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 29 21:20:13.392749 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eadacb3-f6a0-4781-a6aa-a96d261dd669 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:13.486314 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:13.499659 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eadacb3-f6a0-4781-a6aa-a96d261dd669 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:13.531161 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0e0b0b3-181b-41b7-adf5-0b4a48332c3f None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:13.542417 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ff9ad85-bc18-47d3-aae5-a708bf4a634b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:13.543070 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ff9ad85-bc18-47d3-aae5-a708bf4a634b None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:13.543659 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ff9ad85-bc18-47d3-aae5-a708bf4a634b None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:13.544009 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ff9ad85-bc18-47d3-aae5-a708bf4a634b None admin] PATH_INFO: `/v3/roles/member` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:13.555245 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ff9ad85-bc18-47d3-aae5-a708bf4a634b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 21:20:13.559971 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-7ff9ad85-bc18-47d3-aae5-a708bf4a634b None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 29 21:20:13.559971 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:13.559971 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:13.559971 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:13.559971 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:13.559971 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:13.559971 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:13.559971 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:13.559971 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:13.559971 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:13.559971 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:13.559971 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:13.559971 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 21:20:13.559971 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 21:20:13.559971 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 21:20:13.559971 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 21:20:13.560847 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 21:20:13.560847 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 21:20:13.560847 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 21:20:13.560847 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 21:20:13.560847 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:13.560847 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:13.560847 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:13.560847 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:13.560847 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:13.560847 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:13.560847 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:13.560847 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:13.560847 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:13.560847 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:13.560847 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:13.560847 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:13.561666 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:13.561666 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:13.561666 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:13.561666 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:13.561666 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 21:20:13.561666 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 21:20:13.561666 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 21:20:13.561666 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 21:20:13.561666 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 21:20:13.561666 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 21:20:13.561666 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 29 21:20:13.561666 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:20:13.561666 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 149/301] 10.222.0.90 () {56 vars in 1099 bytes} [Mon Aug 29 21:20:13 2022] GET /identity/v3/roles/member => generated 84 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:20:13.562369 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ff9ad85-bc18-47d3-aae5-a708bf4a634b None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:13.643653 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0e0b0b3-181b-41b7-adf5-0b4a48332c3f None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:13.668575 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ff9ad85-bc18-47d3-aae5-a708bf4a634b None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:13.679705 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5c1a5ca-1be7-4891-96c1-5baa16bcfad9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:13.680339 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c1a5ca-1be7-4891-96c1-5baa16bcfad9 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:13.680623 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c1a5ca-1be7-4891-96c1-5baa16bcfad9 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:13.680895 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c1a5ca-1be7-4891-96c1-5baa16bcfad9 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:13.692322 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5c1a5ca-1be7-4891-96c1-5baa16bcfad9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 21:20:13.697637 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-f5c1a5ca-1be7-4891-96c1-5baa16bcfad9 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 29 21:20:13.697637 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:13.697637 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:13.697637 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:13.697637 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:13.697637 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:13.697637 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:13.697637 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:13.697637 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:13.697637 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:13.697637 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:13.697637 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:13.697637 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 21:20:13.697637 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 21:20:13.697637 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 21:20:13.697637 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 21:20:13.698565 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 21:20:13.698565 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 21:20:13.698565 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 21:20:13.698565 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 21:20:13.698565 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:13.698565 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:13.698565 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:13.698565 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:13.698565 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:13.698565 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:13.698565 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:13.698565 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:13.698565 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:13.698565 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:13.698565 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:13.698565 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:13.702580 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:13.702580 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:13.702580 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:13.702580 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:13.702580 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 21:20:13.702580 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 21:20:13.702580 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 21:20:13.702580 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 21:20:13.702580 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 21:20:13.702580 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 21:20:13.702580 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 29 21:20:13.702580 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:20:13.702580 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 153/302] 10.222.0.90 () {56 vars in 1099 bytes} [Mon Aug 29 21:20:13 2022] GET /identity/v3/roles/reader => generated 84 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:20:13.702580 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5c1a5ca-1be7-4891-96c1-5baa16bcfad9 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:13.707211 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-228e5afb-358c-41c0-b42c-e12fd0e27c3a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:13.707925 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-228e5afb-358c-41c0-b42c-e12fd0e27c3a None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:13.708207 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-228e5afb-358c-41c0-b42c-e12fd0e27c3a None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:13.708472 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-228e5afb-358c-41c0-b42c-e12fd0e27c3a None admin] PATH_INFO: `/v3/domains/default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:13.709615 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-228e5afb-358c-41c0-b42c-e12fd0e27c3a None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:13.711102 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-228e5afb-358c-41c0-b42c-e12fd0e27c3a None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:13.712670 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 150/303] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:20:13 2022] GET /identity/v3/domains/default => generated 199 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:13.713421 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-228e5afb-358c-41c0-b42c-e12fd0e27c3a None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:13.732833 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:13.770487 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:13.818599 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5c1a5ca-1be7-4891-96c1-5baa16bcfad9 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:13.824307 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-228e5afb-358c-41c0-b42c-e12fd0e27c3a None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:13.858516 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-183315e8-131f-4ff4-a83e-2e0e1fa22c10 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:13.859407 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-183315e8-131f-4ff4-a83e-2e0e1fa22c10 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:13.860084 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-183315e8-131f-4ff4-a83e-2e0e1fa22c10 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:13.860084 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-183315e8-131f-4ff4-a83e-2e0e1fa22c10 None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:13.860766 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-183315e8-131f-4ff4-a83e-2e0e1fa22c10 None admin] RBAC: Adding query filter params (name=member) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:13.861068 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-183315e8-131f-4ff4-a83e-2e0e1fa22c10 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:13.862476 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-183315e8-131f-4ff4-a83e-2e0e1fa22c10 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:13.868679 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9be446dc-b22e-49c8-9e8a-b5579852339c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:13.869383 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9be446dc-b22e-49c8-9e8a-b5579852339c None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:13.869838 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9be446dc-b22e-49c8-9e8a-b5579852339c None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:13.870261 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9be446dc-b22e-49c8-9e8a-b5579852339c None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:13.870929 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9be446dc-b22e-49c8-9e8a-b5579852339c None admin] RBAC: Adding query filter params (name=reader) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:13.871231 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9be446dc-b22e-49c8-9e8a-b5579852339c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:13.872593 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9be446dc-b22e-49c8-9e8a-b5579852339c None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:13.872898 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 154/304] 10.222.0.90 () {56 vars in 1101 bytes} [Mon Aug 29 21:20:13 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:13.874012 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-183315e8-131f-4ff4-a83e-2e0e1fa22c10 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:13.886878 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 151/305] 10.222.0.90 () {56 vars in 1101 bytes} [Mon Aug 29 21:20:13 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:13.888156 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9be446dc-b22e-49c8-9e8a-b5579852339c None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:13.979475 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:13.998346 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-183315e8-131f-4ff4-a83e-2e0e1fa22c10 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:14.003225 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9be446dc-b22e-49c8-9e8a-b5579852339c None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:14.016424 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:14.034723 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a91d4bf4-a3e9-4c33-b57c-e41c37c70e47 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:14.035947 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a91d4bf4-a3e9-4c33-b57c-e41c37c70e47 None admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:14.036224 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a91d4bf4-a3e9-4c33-b57c-e41c37c70e47 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:14.036494 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a91d4bf4-a3e9-4c33-b57c-e41c37c70e47 None admin] PATH_INFO: `/v3/groups` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:14.037172 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a91d4bf4-a3e9-4c33-b57c-e41c37c70e47 None admin] RBAC: Authorizing `identity:create_group()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:14.038584 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a91d4bf4-a3e9-4c33-b57c-e41c37c70e47 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:14.040139 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7e60d3b-aadc-4bfc-96e1-24dfed4544d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:14.040786 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7e60d3b-aadc-4bfc-96e1-24dfed4544d1 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:14.041177 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7e60d3b-aadc-4bfc-96e1-24dfed4544d1 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:14.041457 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7e60d3b-aadc-4bfc-96e1-24dfed4544d1 None admin] PATH_INFO: `/v3/users/ff1749b05c364ca199eea9870871e691` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:14.042752 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7e60d3b-aadc-4bfc-96e1-24dfed4544d1 None admin] RBAC: Authorizing `identity:get_user(user_id=ff1749b05c364ca199eea9870871e691)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:14.044311 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7e60d3b-aadc-4bfc-96e1-24dfed4544d1 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:14.046002 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 152/306] 10.222.0.90 () {56 vars in 1177 bytes} [Mon Aug 29 21:20:13 2022] GET /identity/v3/users/ff1749b05c364ca199eea9870871e691 => generated 293 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:14.046823 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7e60d3b-aadc-4bfc-96e1-24dfed4544d1 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:14.059538 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 155/307] 10.222.0.90 () {60 vars in 1134 bytes} [Mon Aug 29 21:20:13 2022] POST /identity/v3/groups => generated 229 bytes in 186 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:20:14.060614 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a91d4bf4-a3e9-4c33-b57c-e41c37c70e47 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:14.175197 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7e60d3b-aadc-4bfc-96e1-24dfed4544d1 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:14.197735 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a91d4bf4-a3e9-4c33-b57c-e41c37c70e47 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:14.215935 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2b2e29a-1bb1-413c-8aa3-584a0c135a76 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:14.216592 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2b2e29a-1bb1-413c-8aa3-584a0c135a76 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:14.216875 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2b2e29a-1bb1-413c-8aa3-584a0c135a76 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:14.217152 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2b2e29a-1bb1-413c-8aa3-584a0c135a76 None admin] PATH_INFO: `/v3/users/362070d0dbc049258a4ff352e5f937aa` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:14.239489 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2b2e29a-1bb1-413c-8aa3-584a0c135a76 None admin] RBAC: Authorizing `identity:get_user(user_id=362070d0dbc049258a4ff352e5f937aa)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:14.240960 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2b2e29a-1bb1-413c-8aa3-584a0c135a76 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:14.242870 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 153/308] 10.222.0.90 () {56 vars in 1177 bytes} [Mon Aug 29 21:20:14 2022] GET /identity/v3/users/362070d0dbc049258a4ff352e5f937aa => generated 293 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:14.248688 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2b2e29a-1bb1-413c-8aa3-584a0c135a76 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:14.251138 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93d4031d-1c44-4738-8fd4-f36b8257c9a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:14.252097 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93d4031d-1c44-4738-8fd4-f36b8257c9a7 None admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:14.252560 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93d4031d-1c44-4738-8fd4-f36b8257c9a7 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:14.253056 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93d4031d-1c44-4738-8fd4-f36b8257c9a7 None admin] PATH_INFO: `/v3/system/users/ff1749b05c364ca199eea9870871e691/roles/02e242d5520740d4b77a5f4008c7eb57` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:14.255142 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93d4031d-1c44-4738-8fd4-f36b8257c9a7 None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=ff1749b05c364ca199eea9870871e691, role_id=02e242d5520740d4b77a5f4008c7eb57)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:14.255255 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:14.256533 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93d4031d-1c44-4738-8fd4-f36b8257c9a7 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:14.267474 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 156/309] 10.222.0.90 () {58 vars in 1334 bytes} [Mon Aug 29 21:20:14 2022] PUT /identity/v3/system/users/ff1749b05c364ca199eea9870871e691/roles/02e242d5520740d4b77a5f4008c7eb57 => generated 0 bytes in 207 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:20:14.377280 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2b2e29a-1bb1-413c-8aa3-584a0c135a76 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:14.393597 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aab08c12-b2cd-4a2f-8560-356f31ba08e7 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:14.393927 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aab08c12-b2cd-4a2f-8560-356f31ba08e7 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:14.394194 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aab08c12-b2cd-4a2f-8560-356f31ba08e7 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:14.396551 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 157/310] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:20:14 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 29 21:20:14.402129 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08548a5d-3603-4a38-9125-dd4af9bbfbc5 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:14.402441 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08548a5d-3603-4a38-9125-dd4af9bbfbc5 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:14.402690 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08548a5d-3603-4a38-9125-dd4af9bbfbc5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:14.422415 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f7e778b-4258-4851-abf9-479900d89df9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:14.423041 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f7e778b-4258-4851-abf9-479900d89df9 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:14.423321 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f7e778b-4258-4851-abf9-479900d89df9 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:14.423589 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f7e778b-4258-4851-abf9-479900d89df9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:14.424319 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f7e778b-4258-4851-abf9-479900d89df9 None admin] RBAC: Adding query filter params (role.id=414a14ad48a345f89fd1045f5d8fd16c, user.id=362070d0dbc049258a4ff352e5f937aa, scope.system=all) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:14.424625 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f7e778b-4258-4851-abf9-479900d89df9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:14.425679 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-08548a5d-3603-4a38-9125-dd4af9bbfbc5 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:20:14.426536 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f7e778b-4258-4851-abf9-479900d89df9 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:14.426978 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-08548a5d-3603-4a38-9125-dd4af9bbfbc5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:20:14.438439 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 154/311] 10.222.0.90 () {56 vars in 1308 bytes} [Mon Aug 29 21:20:14 2022] GET /identity/v3/role_assignments?user.id=362070d0dbc049258a4ff352e5f937aa&scope.system=all&role.id=414a14ad48a345f89fd1045f5d8fd16c => generated 226 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:20:14.471017 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:14.511035 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 158/312] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:20:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 29 21:20:14.787396 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f7e778b-4258-4851-abf9-479900d89df9 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:14.858819 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:14.895301 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f7e778b-4258-4851-abf9-479900d89df9 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:14.932200 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baa38a57-6acd-4a7c-817f-7ecf94b4ac6e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:14.932851 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa38a57-6acd-4a7c-817f-7ecf94b4ac6e None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:14.933132 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa38a57-6acd-4a7c-817f-7ecf94b4ac6e None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:14.933405 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa38a57-6acd-4a7c-817f-7ecf94b4ac6e None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:14.944074 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baa38a57-6acd-4a7c-817f-7ecf94b4ac6e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 21:20:14.948930 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-baa38a57-6acd-4a7c-817f-7ecf94b4ac6e None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 29 21:20:14.948930 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:14.948930 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:14.948930 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:14.948930 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:14.948930 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:14.948930 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:14.948930 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:14.948930 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:14.948930 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:14.948930 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:14.948930 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:14.948930 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 21:20:14.948930 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 21:20:14.948930 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 21:20:14.948930 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 21:20:14.949850 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 21:20:14.949850 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 21:20:14.949850 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 21:20:14.949850 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 21:20:14.949850 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:14.949850 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:14.949850 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:14.949850 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:14.949850 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:14.949850 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:14.949850 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:14.949850 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:14.949850 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:14.949850 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:14.949850 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:14.949850 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:14.950666 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:14.950666 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:14.950666 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:14.950666 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:14.950666 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 21:20:14.950666 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 21:20:14.950666 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 21:20:14.950666 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 21:20:14.950666 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 21:20:14.950666 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 21:20:14.950666 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 29 21:20:14.950666 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:20:14.950666 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 155/313] 10.222.0.90 () {56 vars in 1099 bytes} [Mon Aug 29 21:20:14 2022] GET /identity/v3/roles/reader => generated 84 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:20:14.956148 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08548a5d-3603-4a38-9125-dd4af9bbfbc5 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:15.060884 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08548a5d-3603-4a38-9125-dd4af9bbfbc5 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:15.097007 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8e954e8-818e-487e-b2e8-f1284354f381 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:15.098131 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8e954e8-818e-487e-b2e8-f1284354f381 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:15.098734 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8e954e8-818e-487e-b2e8-f1284354f381 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:15.099234 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8e954e8-818e-487e-b2e8-f1284354f381 None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:15.100102 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8e954e8-818e-487e-b2e8-f1284354f381 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:15.100605 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8e954e8-818e-487e-b2e8-f1284354f381 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:15.102206 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8e954e8-818e-487e-b2e8-f1284354f381 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:15.110880 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 159/314] 10.222.0.90 () {56 vars in 1101 bytes} [Mon Aug 29 21:20:14 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:15.121829 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baa38a57-6acd-4a7c-817f-7ecf94b4ac6e None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:15.224514 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baa38a57-6acd-4a7c-817f-7ecf94b4ac6e None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:15.252567 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:15.261625 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-030fc87c-5966-4271-ba2d-79b07be48a6b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:15.262309 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-030fc87c-5966-4271-ba2d-79b07be48a6b None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:15.262598 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-030fc87c-5966-4271-ba2d-79b07be48a6b None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:15.262880 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-030fc87c-5966-4271-ba2d-79b07be48a6b None admin] PATH_INFO: `/v3/users/0e9ebcc726ca470c99dd9b40ff8732d7` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:15.264031 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-030fc87c-5966-4271-ba2d-79b07be48a6b None admin] RBAC: Authorizing `identity:get_user(user_id=0e9ebcc726ca470c99dd9b40ff8732d7)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:15.265729 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-030fc87c-5966-4271-ba2d-79b07be48a6b None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:15.267710 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 156/315] 10.222.0.90 () {56 vars in 1177 bytes} [Mon Aug 29 21:20:15 2022] GET /identity/v3/users/0e9ebcc726ca470c99dd9b40ff8732d7 => generated 297 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:15.274075 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8e954e8-818e-487e-b2e8-f1284354f381 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:15.310099 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:15.386278 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8e954e8-818e-487e-b2e8-f1284354f381 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:15.423164 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33a75bfe-aef1-42ab-a575-5173721f5544 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:15.423779 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a75bfe-aef1-42ab-a575-5173721f5544 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:15.424062 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a75bfe-aef1-42ab-a575-5173721f5544 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:15.424337 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a75bfe-aef1-42ab-a575-5173721f5544 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:15.498071 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33a75bfe-aef1-42ab-a575-5173721f5544 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:15.499543 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33a75bfe-aef1-42ab-a575-5173721f5544 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:15.531242 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:15.605818 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 160/316] 10.222.0.90 () {58 vars in 1282 bytes} [Mon Aug 29 21:20:15 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 332 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 29 21:20:15.611224 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-030fc87c-5966-4271-ba2d-79b07be48a6b None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:15.716389 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-030fc87c-5966-4271-ba2d-79b07be48a6b None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:15.752794 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5144d413-e54b-4c54-86a3-b132c166dbbe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:15.753438 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5144d413-e54b-4c54-86a3-b132c166dbbe None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:15.753723 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5144d413-e54b-4c54-86a3-b132c166dbbe None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:15.754060 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5144d413-e54b-4c54-86a3-b132c166dbbe None admin] PATH_INFO: `/v3/projects/f6d8dbb619904d8889100044bad1bef6` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:15.755353 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5144d413-e54b-4c54-86a3-b132c166dbbe None admin] RBAC: Authorizing `identity:get_project(project_id=f6d8dbb619904d8889100044bad1bef6)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:15.756792 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5144d413-e54b-4c54-86a3-b132c166dbbe None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:15.758717 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 157/317] 10.222.0.90 () {56 vars in 1186 bytes} [Mon Aug 29 21:20:15 2022] GET /identity/v3/projects/f6d8dbb619904d8889100044bad1bef6 => generated 302 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:15.764385 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33a75bfe-aef1-42ab-a575-5173721f5544 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:15.875177 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33a75bfe-aef1-42ab-a575-5173721f5544 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:15.906512 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:15.912174 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26ccee50-fb37-43d8-a11d-26137169275d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:15.912798 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26ccee50-fb37-43d8-a11d-26137169275d None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:15.913116 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26ccee50-fb37-43d8-a11d-26137169275d None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:15.913380 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26ccee50-fb37-43d8-a11d-26137169275d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:15.914207 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26ccee50-fb37-43d8-a11d-26137169275d None admin] RBAC: Adding query filter params (role.id=414a14ad48a345f89fd1045f5d8fd16c, scope.project.id=f6d8dbb619904d8889100044bad1bef6, user.id=0e9ebcc726ca470c99dd9b40ff8732d7) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:15.914876 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26ccee50-fb37-43d8-a11d-26137169275d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:15.916684 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26ccee50-fb37-43d8-a11d-26137169275d None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:15.923652 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 161/318] 10.222.0.90 () {56 vars in 1374 bytes} [Mon Aug 29 21:20:15 2022] GET /identity/v3/role_assignments?user.id=0e9ebcc726ca470c99dd9b40ff8732d7&scope.project.id=f6d8dbb619904d8889100044bad1bef6&role.id=414a14ad48a345f89fd1045f5d8fd16c => generated 608 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:16.148396 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43fed03b-50b0-431d-a21d-709847af6afc None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:16.148910 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43fed03b-50b0-431d-a21d-709847af6afc None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:16.149048 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43fed03b-50b0-431d-a21d-709847af6afc None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:16.150468 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 158/319] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:20:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:20:16.155683 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ffbc2e6-a096-4987-9f31-fb473fdb318a None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:16.155973 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ffbc2e6-a096-4987-9f31-fb473fdb318a None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:16.156247 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ffbc2e6-a096-4987-9f31-fb473fdb318a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:16.174859 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-3ffbc2e6-a096-4987-9f31-fb473fdb318a None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:20:16.176217 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-3ffbc2e6-a096-4987-9f31-fb473fdb318a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:20:16.246424 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 162/320] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:20:16 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 21:20:16.336613 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20330cc9-fe82-4017-affd-17c4c52502fa None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:16.336917 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20330cc9-fe82-4017-affd-17c4c52502fa None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:16.337195 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20330cc9-fe82-4017-affd-17c4c52502fa None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:16.338695 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 159/321] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:20:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:20:16.343579 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7d4d93a-2df3-4217-9b3d-e68ffe93a3a1 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:16.344020 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7d4d93a-2df3-4217-9b3d-e68ffe93a3a1 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:16.344175 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7d4d93a-2df3-4217-9b3d-e68ffe93a3a1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:16.363974 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-a7d4d93a-2df3-4217-9b3d-e68ffe93a3a1 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:20:16.365555 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-a7d4d93a-2df3-4217-9b3d-e68ffe93a3a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:20:16.443347 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 163/322] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:20:16 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 21:20:16.459384 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f938abc-2615-422a-98d5-f0afc7fd8a88 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:16.459663 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f938abc-2615-422a-98d5-f0afc7fd8a88 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:16.459936 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f938abc-2615-422a-98d5-f0afc7fd8a88 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:16.461413 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 160/323] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:20:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:20:16.466437 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65de1060-bd32-4cec-a2e3-f5793249d27b None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:16.466733 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65de1060-bd32-4cec-a2e3-f5793249d27b None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:16.467012 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65de1060-bd32-4cec-a2e3-f5793249d27b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:16.486167 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-65de1060-bd32-4cec-a2e3-f5793249d27b None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:20:16.487357 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-65de1060-bd32-4cec-a2e3-f5793249d27b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:20:16.521057 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f938abc-2615-422a-98d5-f0afc7fd8a88 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:16.541600 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:16.559383 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 164/324] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:20:16 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 21:20:16.591772 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:16.633260 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f938abc-2615-422a-98d5-f0afc7fd8a88 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:16.669588 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31df98f6-dfe3-40ff-83cf-d8d08337b736 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:16.670307 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31df98f6-dfe3-40ff-83cf-d8d08337b736 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:16.670649 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31df98f6-dfe3-40ff-83cf-d8d08337b736 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:16.670923 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31df98f6-dfe3-40ff-83cf-d8d08337b736 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:16.672022 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31df98f6-dfe3-40ff-83cf-d8d08337b736 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:16.673518 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31df98f6-dfe3-40ff-83cf-d8d08337b736 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:16.677816 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 161/325] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:20:16 2022] GET /identity/v3/domains/default => generated 199 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:20:16.680683 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65de1060-bd32-4cec-a2e3-f5793249d27b None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:16.710854 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31df98f6-dfe3-40ff-83cf-d8d08337b736 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:16.785818 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65de1060-bd32-4cec-a2e3-f5793249d27b None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:16.824209 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cf70b69-a5b9-4c62-a89e-0f67c6b22e2f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:16.824826 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cf70b69-a5b9-4c62-a89e-0f67c6b22e2f None admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:16.824968 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31df98f6-dfe3-40ff-83cf-d8d08337b736 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:16.825166 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cf70b69-a5b9-4c62-a89e-0f67c6b22e2f None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:16.825369 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cf70b69-a5b9-4c62-a89e-0f67c6b22e2f None admin] PATH_INFO: `/v3/groups` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:16.826040 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cf70b69-a5b9-4c62-a89e-0f67c6b22e2f None admin] RBAC: Authorizing `identity:create_group()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:16.828541 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cf70b69-a5b9-4c62-a89e-0f67c6b22e2f None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:16.845720 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 165/326] 10.222.0.90 () {60 vars in 1134 bytes} [Mon Aug 29 21:20:16 2022] POST /identity/v3/groups => generated 226 bytes in 165 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:20:16.846894 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cf70b69-a5b9-4c62-a89e-0f67c6b22e2f None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:16.857823 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:16.866270 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d4b74f4-d627-4048-a9cc-15c535f3368f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:16.866940 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d4b74f4-d627-4048-a9cc-15c535f3368f None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:16.867220 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d4b74f4-d627-4048-a9cc-15c535f3368f None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:16.867516 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d4b74f4-d627-4048-a9cc-15c535f3368f None admin] PATH_INFO: `/v3/roles/member` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:16.880055 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d4b74f4-d627-4048-a9cc-15c535f3368f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 21:20:16.885234 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-7d4b74f4-d627-4048-a9cc-15c535f3368f None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 29 21:20:16.885234 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:16.885234 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:16.885234 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:16.885234 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:16.885234 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:16.885234 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:16.885234 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:16.885234 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:16.885234 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:16.885234 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:16.885234 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:16.885234 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 21:20:16.885234 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 21:20:16.885234 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 21:20:16.885234 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 21:20:16.886266 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 21:20:16.886266 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 21:20:16.886266 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 21:20:16.886266 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 21:20:16.886266 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:16.886266 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:16.886266 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:16.886266 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:16.886266 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:16.886266 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:16.886266 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:16.886266 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:16.886266 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:16.886266 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:16.886266 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:16.886266 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:16.891266 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:16.891266 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:16.891266 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:16.891266 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:16.891266 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 21:20:16.891266 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 21:20:16.891266 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 21:20:16.891266 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 21:20:16.891266 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 21:20:16.891266 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 21:20:16.891266 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 29 21:20:16.891266 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:20:16.891266 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 162/327] 10.222.0.90 () {56 vars in 1099 bytes} [Mon Aug 29 21:20:16 2022] GET /identity/v3/roles/member => generated 84 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 29 21:20:16.892595 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d4b74f4-d627-4048-a9cc-15c535f3368f None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:16.913933 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:16.976651 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cf70b69-a5b9-4c62-a89e-0f67c6b22e2f None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:17.016701 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c076daad-79ac-4bb3-8722-53c2721fdbb7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:17.017348 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c076daad-79ac-4bb3-8722-53c2721fdbb7 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:17.017632 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c076daad-79ac-4bb3-8722-53c2721fdbb7 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:17.017939 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c076daad-79ac-4bb3-8722-53c2721fdbb7 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:17.022659 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d4b74f4-d627-4048-a9cc-15c535f3368f None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:17.030768 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c076daad-79ac-4bb3-8722-53c2721fdbb7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 21:20:17.036956 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-c076daad-79ac-4bb3-8722-53c2721fdbb7 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 29 21:20:17.036956 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:17.036956 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:17.036956 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:17.036956 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:17.036956 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:17.036956 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:17.036956 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:17.036956 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:17.036956 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:17.036956 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:17.036956 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:17.036956 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 21:20:17.036956 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 21:20:17.036956 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 21:20:17.036956 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 21:20:17.037951 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 21:20:17.037951 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 21:20:17.037951 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 21:20:17.037951 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 21:20:17.037951 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:17.037951 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:17.037951 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:17.037951 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:17.037951 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:17.037951 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:17.037951 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:17.037951 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:17.037951 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:17.037951 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:17.037951 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:17.037951 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:17.038773 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:17.038773 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:17.038773 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:17.038773 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:17.038773 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 21:20:17.038773 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 21:20:17.038773 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 21:20:17.038773 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 21:20:17.038773 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 21:20:17.038773 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 21:20:17.038773 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 29 21:20:17.038773 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:20:17.038773 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 166/328] 10.222.0.90 () {56 vars in 1099 bytes} [Mon Aug 29 21:20:16 2022] GET /identity/v3/roles/reader => generated 84 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:20:17.045518 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c076daad-79ac-4bb3-8722-53c2721fdbb7 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:17.063898 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30373502-dab3-4c32-abe9-f65ce3d27b44 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:17.066880 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30373502-dab3-4c32-abe9-f65ce3d27b44 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:17.066880 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30373502-dab3-4c32-abe9-f65ce3d27b44 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:17.066880 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30373502-dab3-4c32-abe9-f65ce3d27b44 None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:17.066880 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30373502-dab3-4c32-abe9-f65ce3d27b44 None admin] RBAC: Adding query filter params (name=member) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:17.066880 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30373502-dab3-4c32-abe9-f65ce3d27b44 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:17.068293 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30373502-dab3-4c32-abe9-f65ce3d27b44 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:17.078027 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 163/329] 10.222.0.90 () {56 vars in 1101 bytes} [Mon Aug 29 21:20:16 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:20:17.085490 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30373502-dab3-4c32-abe9-f65ce3d27b44 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:17.161356 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:17.167072 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c076daad-79ac-4bb3-8722-53c2721fdbb7 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:17.180845 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:17.214490 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47dac772-c528-4946-8916-9abdd5adc37f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:17.215119 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47dac772-c528-4946-8916-9abdd5adc37f None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:17.215401 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47dac772-c528-4946-8916-9abdd5adc37f None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:17.215685 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47dac772-c528-4946-8916-9abdd5adc37f None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:17.216354 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47dac772-c528-4946-8916-9abdd5adc37f None admin] RBAC: Adding query filter params (name=reader) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:17.216654 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47dac772-c528-4946-8916-9abdd5adc37f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:17.218892 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47dac772-c528-4946-8916-9abdd5adc37f None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:17.220744 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30373502-dab3-4c32-abe9-f65ce3d27b44 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:17.228604 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 167/330] 10.222.0.90 () {56 vars in 1101 bytes} [Mon Aug 29 21:20:17 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:17.235344 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47dac772-c528-4946-8916-9abdd5adc37f None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:17.268628 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3619833-e7ee-44be-a994-4021ccda5197 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:17.270177 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3619833-e7ee-44be-a994-4021ccda5197 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:17.270700 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3619833-e7ee-44be-a994-4021ccda5197 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:17.271119 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3619833-e7ee-44be-a994-4021ccda5197 None admin] PATH_INFO: `/v3/users/ff1749b05c364ca199eea9870871e691` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:17.272646 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3619833-e7ee-44be-a994-4021ccda5197 None admin] RBAC: Authorizing `identity:get_user(user_id=ff1749b05c364ca199eea9870871e691)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:17.275099 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3619833-e7ee-44be-a994-4021ccda5197 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:17.277178 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 164/331] 10.222.0.90 () {56 vars in 1177 bytes} [Mon Aug 29 21:20:17 2022] GET /identity/v3/users/ff1749b05c364ca199eea9870871e691 => generated 293 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:17.283252 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3619833-e7ee-44be-a994-4021ccda5197 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:17.365391 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47dac772-c528-4946-8916-9abdd5adc37f None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:17.404101 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-968dd8b0-59d8-4c39-a342-b73b39426c36 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:17.404750 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-968dd8b0-59d8-4c39-a342-b73b39426c36 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:17.405034 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-968dd8b0-59d8-4c39-a342-b73b39426c36 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:17.405303 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-968dd8b0-59d8-4c39-a342-b73b39426c36 None admin] PATH_INFO: `/v3/users/362070d0dbc049258a4ff352e5f937aa` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:17.406563 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-968dd8b0-59d8-4c39-a342-b73b39426c36 None admin] RBAC: Authorizing `identity:get_user(user_id=362070d0dbc049258a4ff352e5f937aa)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:17.408140 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-968dd8b0-59d8-4c39-a342-b73b39426c36 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:17.410506 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3619833-e7ee-44be-a994-4021ccda5197 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:17.413937 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 168/332] 10.222.0.90 () {56 vars in 1177 bytes} [Mon Aug 29 21:20:17 2022] GET /identity/v3/users/362070d0dbc049258a4ff352e5f937aa => generated 293 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:20:17.416165 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-968dd8b0-59d8-4c39-a342-b73b39426c36 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:17.436538 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:17.453832 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edf4c950-c1c8-47bc-a560-03a2a980b900 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:17.454512 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edf4c950-c1c8-47bc-a560-03a2a980b900 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:17.454819 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edf4c950-c1c8-47bc-a560-03a2a980b900 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:17.455138 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edf4c950-c1c8-47bc-a560-03a2a980b900 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:17.457089 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edf4c950-c1c8-47bc-a560-03a2a980b900 None admin] RBAC: Adding query filter params (role.id=02e242d5520740d4b77a5f4008c7eb57, user.id=ff1749b05c364ca199eea9870871e691, scope.system=all) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:17.457580 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edf4c950-c1c8-47bc-a560-03a2a980b900 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:17.458874 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:17.461298 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edf4c950-c1c8-47bc-a560-03a2a980b900 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:17.481966 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 165/333] 10.222.0.90 () {56 vars in 1308 bytes} [Mon Aug 29 21:20:17 2022] GET /identity/v3/role_assignments?user.id=ff1749b05c364ca199eea9870871e691&scope.system=all&role.id=02e242d5520740d4b77a5f4008c7eb57 => generated 510 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:17.542000 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-968dd8b0-59d8-4c39-a342-b73b39426c36 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:17.582685 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13294284-79f4-43c8-b7d9-966ae1dfc75b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:17.583388 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13294284-79f4-43c8-b7d9-966ae1dfc75b None admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:17.583661 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13294284-79f4-43c8-b7d9-966ae1dfc75b None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:17.583962 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13294284-79f4-43c8-b7d9-966ae1dfc75b None admin] PATH_INFO: `/v3/system/users/362070d0dbc049258a4ff352e5f937aa/roles/414a14ad48a345f89fd1045f5d8fd16c` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:17.585607 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13294284-79f4-43c8-b7d9-966ae1dfc75b None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=362070d0dbc049258a4ff352e5f937aa, role_id=414a14ad48a345f89fd1045f5d8fd16c)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:17.587072 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13294284-79f4-43c8-b7d9-966ae1dfc75b None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:17.595765 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 169/334] 10.222.0.90 () {58 vars in 1334 bytes} [Mon Aug 29 21:20:17 2022] PUT /identity/v3/system/users/362070d0dbc049258a4ff352e5f937aa/roles/414a14ad48a345f89fd1045f5d8fd16c => generated 0 bytes in 180 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 21:20:19.086584 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68b2d9b6-ec9a-4fb4-b015-300fa267491f None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:19.087078 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68b2d9b6-ec9a-4fb4-b015-300fa267491f None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:19.087180 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68b2d9b6-ec9a-4fb4-b015-300fa267491f None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:19.088618 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 166/335] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:20:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:20:19.097457 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6835a90-cfdb-42c9-a82b-cb9fe9137494 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:19.098417 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6835a90-cfdb-42c9-a82b-cb9fe9137494 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:19.098887 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6835a90-cfdb-42c9-a82b-cb9fe9137494 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:19.122000 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-b6835a90-cfdb-42c9-a82b-cb9fe9137494 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:20:19.123504 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-b6835a90-cfdb-42c9-a82b-cb9fe9137494 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:20:19.200429 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 170/336] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:20:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 21:20:19.382372 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cbdf51f-e3fc-46cf-87ac-cdc79212942f None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:19.382976 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cbdf51f-e3fc-46cf-87ac-cdc79212942f None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:19.383454 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cbdf51f-e3fc-46cf-87ac-cdc79212942f None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:19.385236 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 167/337] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:20:19 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:20:19.396940 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d7d6ef3-72d2-4aac-ba9f-474287607e1d None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:19.397574 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d7d6ef3-72d2-4aac-ba9f-474287607e1d None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:19.398094 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d7d6ef3-72d2-4aac-ba9f-474287607e1d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:19.420979 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-7d7d6ef3-72d2-4aac-ba9f-474287607e1d None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:20:19.422590 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-7d7d6ef3-72d2-4aac-ba9f-474287607e1d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:20:19.454027 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:19.502404 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 171/338] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:20:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 21:20:19.686831 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cbdf51f-e3fc-46cf-87ac-cdc79212942f None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:19.827034 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cbdf51f-e3fc-46cf-87ac-cdc79212942f None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:19.874492 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83b66049-e125-467d-a784-beca2090c76d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:19.875140 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83b66049-e125-467d-a784-beca2090c76d None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:19.875418 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83b66049-e125-467d-a784-beca2090c76d None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:19.875692 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83b66049-e125-467d-a784-beca2090c76d None admin] PATH_INFO: `/v3/roles/member` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:19.890706 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83b66049-e125-467d-a784-beca2090c76d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 21:20:19.896583 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-83b66049-e125-467d-a784-beca2090c76d None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 29 21:20:19.896583 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:19.896583 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:19.896583 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:19.896583 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:19.896583 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:19.896583 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:19.896583 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:19.896583 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:19.896583 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:19.896583 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:19.896583 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:19.896583 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 21:20:19.896583 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 21:20:19.896583 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 21:20:19.896583 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 21:20:19.897471 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 21:20:19.897471 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 21:20:19.897471 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 21:20:19.897471 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 21:20:19.897471 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:19.897471 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:19.897471 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:19.897471 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:19.897471 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:19.897471 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:19.897471 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:19.897471 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:19.897471 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:19.897471 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:19.897471 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:19.897471 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:19.898391 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:19.898391 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:19.898391 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:19.898391 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:19.898391 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 21:20:19.898391 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 21:20:19.898391 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 21:20:19.898391 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 21:20:19.898391 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 21:20:19.898391 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 21:20:19.898391 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 29 21:20:19.898391 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:20:19.898391 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 168/339] 10.222.0.90 () {56 vars in 1099 bytes} [Mon Aug 29 21:20:19 2022] GET /identity/v3/roles/member => generated 84 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:20:19.903444 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d7d6ef3-72d2-4aac-ba9f-474287607e1d None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:19.944051 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83b66049-e125-467d-a784-beca2090c76d None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:19.998164 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:20.063637 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83b66049-e125-467d-a784-beca2090c76d None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:20.104241 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e7693ad-e0c0-4a2d-a0c5-c5f48ca4b321 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:20.105076 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e7693ad-e0c0-4a2d-a0c5-c5f48ca4b321 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:20.105529 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e7693ad-e0c0-4a2d-a0c5-c5f48ca4b321 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:20.105967 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e7693ad-e0c0-4a2d-a0c5-c5f48ca4b321 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:20.118399 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e7693ad-e0c0-4a2d-a0c5-c5f48ca4b321 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 21:20:20.125316 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-3e7693ad-e0c0-4a2d-a0c5-c5f48ca4b321 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 29 21:20:20.125316 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:20.125316 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:20.125316 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:20.125316 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:20.125316 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:20.125316 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:20.125316 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:20.125316 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:20.125316 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:20.125316 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:20.125316 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:20.125316 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 21:20:20.125316 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 21:20:20.125316 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 21:20:20.125316 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 21:20:20.126289 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 21:20:20.126289 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 21:20:20.126289 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 21:20:20.126289 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 21:20:20.126289 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:20.126289 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:20.126289 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:20.126289 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:20.126289 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:20.126289 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:20.126289 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:20.126289 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:20.126289 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:20.126289 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:20.126289 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:20.126289 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:20.127367 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:20.127367 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:20.127367 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:20.127367 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:20.127367 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 21:20:20.127367 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 21:20:20.127367 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 21:20:20.127367 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 21:20:20.127367 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 21:20:20.127367 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 21:20:20.127367 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 29 21:20:20.127367 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:20:20.127367 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 169/340] 10.222.0.90 () {56 vars in 1114 bytes} [Mon Aug 29 21:20:19 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 29 21:20:20.134159 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e7693ad-e0c0-4a2d-a0c5-c5f48ca4b321 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:20.148742 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d7d6ef3-72d2-4aac-ba9f-474287607e1d None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:20.201575 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b16aa1f2-68e6-4c54-87c8-4536cab7e845 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:20.202146 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b16aa1f2-68e6-4c54-87c8-4536cab7e845 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:20.202468 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b16aa1f2-68e6-4c54-87c8-4536cab7e845 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:20.202694 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b16aa1f2-68e6-4c54-87c8-4536cab7e845 None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:20.203367 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b16aa1f2-68e6-4c54-87c8-4536cab7e845 None admin] RBAC: Adding query filter params (name=member) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:20.203670 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b16aa1f2-68e6-4c54-87c8-4536cab7e845 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:20.205039 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b16aa1f2-68e6-4c54-87c8-4536cab7e845 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:20.217368 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 172/341] 10.222.0.90 () {56 vars in 1101 bytes} [Mon Aug 29 21:20:19 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 314 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:20.223423 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b16aa1f2-68e6-4c54-87c8-4536cab7e845 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:20.251028 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e7693ad-e0c0-4a2d-a0c5-c5f48ca4b321 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:20.254104 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:20.259237 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:20.291906 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5df2cb1e-177b-48f3-addb-101204c709f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:20.292587 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5df2cb1e-177b-48f3-addb-101204c709f0 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:20.292877 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5df2cb1e-177b-48f3-addb-101204c709f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:20.293161 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5df2cb1e-177b-48f3-addb-101204c709f0 None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:20.293919 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5df2cb1e-177b-48f3-addb-101204c709f0 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:20.294231 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5df2cb1e-177b-48f3-addb-101204c709f0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:20.296768 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5df2cb1e-177b-48f3-addb-101204c709f0 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:20.306901 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 170/342] 10.222.0.90 () {56 vars in 1111 bytes} [Mon Aug 29 21:20:20 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:20.312571 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5df2cb1e-177b-48f3-addb-101204c709f0 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:20.378075 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b16aa1f2-68e6-4c54-87c8-4536cab7e845 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:20.429161 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b5a504d-8f54-4b3b-bcd4-81e650d28f7f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:20.429950 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b5a504d-8f54-4b3b-bcd4-81e650d28f7f None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:20.430266 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b5a504d-8f54-4b3b-bcd4-81e650d28f7f None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:20.430531 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b5a504d-8f54-4b3b-bcd4-81e650d28f7f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:20.430824 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5df2cb1e-177b-48f3-addb-101204c709f0 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:20.470022 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a887cea7-de4b-4341-8c95-3b92e075b80e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:20.470568 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a887cea7-de4b-4341-8c95-3b92e075b80e None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:20.470848 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a887cea7-de4b-4341-8c95-3b92e075b80e None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:20.471121 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a887cea7-de4b-4341-8c95-3b92e075b80e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:20.508689 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:20.509257 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b5a504d-8f54-4b3b-bcd4-81e650d28f7f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:20.510955 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b5a504d-8f54-4b3b-bcd4-81e650d28f7f None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:20.540682 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:20.556054 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a887cea7-de4b-4341-8c95-3b92e075b80e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:20.557534 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a887cea7-de4b-4341-8c95-3b92e075b80e None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:20.626039 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 173/343] 10.222.0.90 () {58 vars in 1282 bytes} [Mon Aug 29 21:20:20 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 403 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Aug 29 21:20:20.627408 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3e4a266-109d-41ab-b52e-41fa18debc50 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:20.627695 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3e4a266-109d-41ab-b52e-41fa18debc50 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:20.628016 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3e4a266-109d-41ab-b52e-41fa18debc50 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:20.629609 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 174/344] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:20:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:20:20.630915 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04491ef0-dad6-446e-9f82-d12f198f9c20 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:20.631109 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04491ef0-dad6-446e-9f82-d12f198f9c20 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:20.631356 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04491ef0-dad6-446e-9f82-d12f198f9c20 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:20.632783 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 175/345] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:20:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:20:20.633487 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04491ef0-dad6-446e-9f82-d12f198f9c20 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:20.671580 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 171/346] 10.222.0.90 () {58 vars in 1282 bytes} [Mon Aug 29 21:20:20 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 359 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 29 21:20:20.672911 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89052b2b-7c68-441e-b3c5-f83ab5f7b1dd None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:20.673237 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89052b2b-7c68-441e-b3c5-f83ab5f7b1dd None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:20.673539 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89052b2b-7c68-441e-b3c5-f83ab5f7b1dd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:20.701388 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-89052b2b-7c68-441e-b3c5-f83ab5f7b1dd None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:20:20.702637 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-89052b2b-7c68-441e-b3c5-f83ab5f7b1dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:20:20.746717 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04491ef0-dad6-446e-9f82-d12f198f9c20 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:20.768579 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:20.773140 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:20.781676 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 172/347] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:20:20 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 21:20:20.784130 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5f22c19-15e4-4eb2-b8c3-010580b070e0 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:20.784654 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5f22c19-15e4-4eb2-b8c3-010580b070e0 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:20.785277 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5f22c19-15e4-4eb2-b8c3-010580b070e0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:20.786377 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7309ece-772b-4d37-b245-a764f91a3fbb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:20.787017 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7309ece-772b-4d37-b245-a764f91a3fbb None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:20.787295 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7309ece-772b-4d37-b245-a764f91a3fbb None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:20.787623 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7309ece-772b-4d37-b245-a764f91a3fbb None admin] PATH_INFO: `/v3/projects/381cc34791e147329798cd454f99f622` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:20.788820 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7309ece-772b-4d37-b245-a764f91a3fbb None admin] RBAC: Authorizing `identity:get_project(project_id=381cc34791e147329798cd454f99f622)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:20.790359 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7309ece-772b-4d37-b245-a764f91a3fbb None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:20.792268 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 176/348] 10.222.0.90 () {56 vars in 1186 bytes} [Mon Aug 29 21:20:20 2022] GET /identity/v3/projects/381cc34791e147329798cd454f99f622 => generated 298 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:20.792856 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7309ece-772b-4d37-b245-a764f91a3fbb None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:20.807005 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-b5f22c19-15e4-4eb2-b8c3-010580b070e0 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:20:20.808526 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-b5f22c19-15e4-4eb2-b8c3-010580b070e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:20:20.893486 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 173/349] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:20:20 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 21:20:20.895274 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5f22c19-15e4-4eb2-b8c3-010580b070e0 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:20.936158 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7309ece-772b-4d37-b245-a764f91a3fbb None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:20.989983 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ba55863-5fa9-4d12-ad19-9d0fcd6d729c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:20.990749 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ba55863-5fa9-4d12-ad19-9d0fcd6d729c None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:20.991028 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ba55863-5fa9-4d12-ad19-9d0fcd6d729c None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:20.991295 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ba55863-5fa9-4d12-ad19-9d0fcd6d729c None admin] PATH_INFO: `/v3/projects/381cc34791e147329798cd454f99f622` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:20.992440 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ba55863-5fa9-4d12-ad19-9d0fcd6d729c None admin] RBAC: Authorizing `identity:get_project(project_id=381cc34791e147329798cd454f99f622)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:20.994086 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ba55863-5fa9-4d12-ad19-9d0fcd6d729c None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:20.996635 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 177/350] 10.222.0.90 () {56 vars in 1186 bytes} [Mon Aug 29 21:20:20 2022] GET /identity/v3/projects/381cc34791e147329798cd454f99f622 => generated 298 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:21.009637 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ba55863-5fa9-4d12-ad19-9d0fcd6d729c None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:21.079765 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5f22c19-15e4-4eb2-b8c3-010580b070e0 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:21.095197 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:21.105493 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:21.119863 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4e47b0e-a6f9-4641-8db0-506b1d54bd7a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:21.120516 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4e47b0e-a6f9-4641-8db0-506b1d54bd7a None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:21.120797 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4e47b0e-a6f9-4641-8db0-506b1d54bd7a None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:21.121073 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4e47b0e-a6f9-4641-8db0-506b1d54bd7a None admin] PATH_INFO: `/v3/groups/792e5c9f64ca4f019cee6fd3fe995744` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:21.127301 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4e47b0e-a6f9-4641-8db0-506b1d54bd7a None admin] RBAC: Authorizing `identity:get_group(group_id=792e5c9f64ca4f019cee6fd3fe995744)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:21.128791 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4e47b0e-a6f9-4641-8db0-506b1d54bd7a None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:21.131517 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 174/351] 10.222.0.90 () {56 vars in 1180 bytes} [Mon Aug 29 21:20:20 2022] GET /identity/v3/groups/792e5c9f64ca4f019cee6fd3fe995744 => generated 226 bytes in 237 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:21.132533 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d7421f-63b1-4341-a8d4-832250d78f5a None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:21.132815 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d7421f-63b1-4341-a8d4-832250d78f5a None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:21.133089 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d7421f-63b1-4341-a8d4-832250d78f5a None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:21.138538 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 175/352] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:20:21 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:20:21.139968 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28d7421f-63b1-4341-a8d4-832250d78f5a None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:21.146825 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ba55863-5fa9-4d12-ad19-9d0fcd6d729c None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:21.186670 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4aa30c3-f425-4585-b9f4-3ed1404c5936 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:21.187312 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4aa30c3-f425-4585-b9f4-3ed1404c5936 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:21.187574 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4aa30c3-f425-4585-b9f4-3ed1404c5936 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:21.187835 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4aa30c3-f425-4585-b9f4-3ed1404c5936 None admin] PATH_INFO: `/v3/groups/792e5c9f64ca4f019cee6fd3fe995744` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:21.188949 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4aa30c3-f425-4585-b9f4-3ed1404c5936 None admin] RBAC: Authorizing `identity:get_group(group_id=792e5c9f64ca4f019cee6fd3fe995744)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:21.190446 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4aa30c3-f425-4585-b9f4-3ed1404c5936 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:21.192556 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 178/353] 10.222.0.90 () {56 vars in 1180 bytes} [Mon Aug 29 21:20:21 2022] GET /identity/v3/groups/792e5c9f64ca4f019cee6fd3fe995744 => generated 226 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:21.193802 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-744743b8-7276-4c25-9353-0ebf350143ab None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:21.194104 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-744743b8-7276-4c25-9353-0ebf350143ab None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:21.194371 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-744743b8-7276-4c25-9353-0ebf350143ab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:21.213072 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-744743b8-7276-4c25-9353-0ebf350143ab None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:20:21.214289 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-744743b8-7276-4c25-9353-0ebf350143ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:20:21.251627 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28d7421f-63b1-4341-a8d4-832250d78f5a None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:21.287375 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 179/354] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:20:21 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 21:20:21.288238 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-744743b8-7276-4c25-9353-0ebf350143ab None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:21.288785 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba1f3b51-0edb-425a-8fbd-1afb6855b45b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:21.289410 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba1f3b51-0edb-425a-8fbd-1afb6855b45b None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:21.289686 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba1f3b51-0edb-425a-8fbd-1afb6855b45b None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:21.290066 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba1f3b51-0edb-425a-8fbd-1afb6855b45b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:21.290799 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba1f3b51-0edb-425a-8fbd-1afb6855b45b None admin] RBAC: Adding query filter params (group.id=792e5c9f64ca4f019cee6fd3fe995744, role.id=02e242d5520740d4b77a5f4008c7eb57, scope.project.id=381cc34791e147329798cd454f99f622) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:21.291099 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba1f3b51-0edb-425a-8fbd-1afb6855b45b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:21.292520 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba1f3b51-0edb-425a-8fbd-1afb6855b45b None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:21.298632 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 176/355] 10.222.0.90 () {56 vars in 1376 bytes} [Mon Aug 29 21:20:21 2022] GET /identity/v3/role_assignments?group.id=792e5c9f64ca4f019cee6fd3fe995744&scope.project.id=381cc34791e147329798cd454f99f622&role.id=02e242d5520740d4b77a5f4008c7eb57 => generated 260 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:20:21.352311 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:21.365678 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba1f3b51-0edb-425a-8fbd-1afb6855b45b None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:21.401188 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-744743b8-7276-4c25-9353-0ebf350143ab None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:21.436993 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfd267b1-c99e-4028-b167-25a3bd7e3d2a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:21.437611 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd267b1-c99e-4028-b167-25a3bd7e3d2a None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:21.437922 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd267b1-c99e-4028-b167-25a3bd7e3d2a None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:21.438197 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd267b1-c99e-4028-b167-25a3bd7e3d2a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:21.438928 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfd267b1-c99e-4028-b167-25a3bd7e3d2a None admin] RBAC: Adding query filter params (group.id=792e5c9f64ca4f019cee6fd3fe995744, role.id=9745586777bf473bab7d887bf25bf236, scope.project.id=381cc34791e147329798cd454f99f622) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:21.439231 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfd267b1-c99e-4028-b167-25a3bd7e3d2a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:21.440706 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfd267b1-c99e-4028-b167-25a3bd7e3d2a None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:21.447431 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 180/356] 10.222.0.90 () {56 vars in 1376 bytes} [Mon Aug 29 21:20:21 2022] GET /identity/v3/role_assignments?group.id=792e5c9f64ca4f019cee6fd3fe995744&scope.project.id=381cc34791e147329798cd454f99f622&role.id=9745586777bf473bab7d887bf25bf236 => generated 260 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:21.448063 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfd267b1-c99e-4028-b167-25a3bd7e3d2a None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:21.448807 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:21.530991 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba1f3b51-0edb-425a-8fbd-1afb6855b45b None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:21.565647 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfd267b1-c99e-4028-b167-25a3bd7e3d2a None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:21.587463 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:21.592992 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a73d88eb-d3bb-4788-b258-8ce99f9c49c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:21.593952 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a73d88eb-d3bb-4788-b258-8ce99f9c49c2 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:21.594411 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a73d88eb-d3bb-4788-b258-8ce99f9c49c2 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:21.594889 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a73d88eb-d3bb-4788-b258-8ce99f9c49c2 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:21.604300 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5edf2fa-4c82-403d-97a9-415d46a58ee6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:21.604870 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5edf2fa-4c82-403d-97a9-415d46a58ee6 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:21.605152 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5edf2fa-4c82-403d-97a9-415d46a58ee6 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:21.605421 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5edf2fa-4c82-403d-97a9-415d46a58ee6 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:21.609215 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a73d88eb-d3bb-4788-b258-8ce99f9c49c2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 21:20:21.616032 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-a73d88eb-d3bb-4788-b258-8ce99f9c49c2 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 29 21:20:21.616032 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:21.616032 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:21.616032 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:21.616032 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:21.616032 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:21.616032 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:21.616032 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:21.616032 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:21.616032 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:21.616032 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:21.616032 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:21.616032 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 21:20:21.616032 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 21:20:21.616032 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 21:20:21.616032 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 21:20:21.617209 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 21:20:21.617209 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 21:20:21.617209 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 21:20:21.617209 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 21:20:21.617209 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:21.617209 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:21.617209 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:21.617209 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:21.617209 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:21.617209 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:21.617209 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:21.617209 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:21.617209 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:21.617209 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:21.617209 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:21.617209 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:21.619378 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:21.619378 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:21.619378 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:21.619378 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:21.619378 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 21:20:21.619378 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 21:20:21.619378 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 21:20:21.619378 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 21:20:21.619378 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 21:20:21.619378 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 21:20:21.619378 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 29 21:20:21.619378 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:20:21.620136 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5edf2fa-4c82-403d-97a9-415d46a58ee6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 21:20:21.621850 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 177/357] 10.222.0.90 () {56 vars in 1099 bytes} [Mon Aug 29 21:20:21 2022] GET /identity/v3/roles/member => generated 84 bytes in 257 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:20:21.625661 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-e5edf2fa-4c82-403d-97a9-415d46a58ee6 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 29 21:20:21.625661 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:21.625661 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:21.625661 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:21.625661 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:21.625661 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:21.625661 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:21.625661 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:21.625661 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:21.625661 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:21.625661 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:21.625661 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:21.625661 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 21:20:21.625661 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 21:20:21.625661 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 21:20:21.625661 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 21:20:21.626814 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 21:20:21.626814 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 21:20:21.626814 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 21:20:21.626814 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 21:20:21.626814 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:21.626814 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:21.626814 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:21.626814 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:21.626814 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:21.626814 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:21.626814 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:21.626814 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:21.626814 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:21.626814 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:21.626814 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:21.626814 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:21.627852 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:21.627852 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:21.627852 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:21.627852 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:21.627852 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 21:20:21.627852 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 21:20:21.627852 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 21:20:21.627852 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 21:20:21.627852 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 21:20:21.627852 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 21:20:21.627852 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 29 21:20:21.627852 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:20:21.627852 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 181/358] 10.222.0.90 () {56 vars in 1099 bytes} [Mon Aug 29 21:20:21 2022] GET /identity/v3/roles/reader => generated 84 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:20:21.630278 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a73d88eb-d3bb-4788-b258-8ce99f9c49c2 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:21.635227 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5edf2fa-4c82-403d-97a9-415d46a58ee6 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:21.764180 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5edf2fa-4c82-403d-97a9-415d46a58ee6 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:21.772476 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a73d88eb-d3bb-4788-b258-8ce99f9c49c2 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:21.784582 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:21.807154 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70be9fac-dc03-46a2-a6d2-d07e2dd59ca5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:21.808282 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70be9fac-dc03-46a2-a6d2-d07e2dd59ca5 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:21.808705 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70be9fac-dc03-46a2-a6d2-d07e2dd59ca5 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:21.809380 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70be9fac-dc03-46a2-a6d2-d07e2dd59ca5 None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:21.810232 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70be9fac-dc03-46a2-a6d2-d07e2dd59ca5 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:21.810688 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70be9fac-dc03-46a2-a6d2-d07e2dd59ca5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:21.813194 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70be9fac-dc03-46a2-a6d2-d07e2dd59ca5 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:21.816028 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1e54ef9-47c0-4e23-8f41-599af440193a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:21.816662 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1e54ef9-47c0-4e23-8f41-599af440193a None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:21.816896 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1e54ef9-47c0-4e23-8f41-599af440193a None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:21.817168 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1e54ef9-47c0-4e23-8f41-599af440193a None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:21.818253 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1e54ef9-47c0-4e23-8f41-599af440193a None admin] RBAC: Adding query filter params (name=member) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:21.818561 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1e54ef9-47c0-4e23-8f41-599af440193a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:21.819955 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1e54ef9-47c0-4e23-8f41-599af440193a None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:21.825666 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 182/359] 10.222.0.90 () {56 vars in 1101 bytes} [Mon Aug 29 21:20:21 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:21.827280 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70be9fac-dc03-46a2-a6d2-d07e2dd59ca5 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:21.833870 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 178/360] 10.222.0.90 () {56 vars in 1101 bytes} [Mon Aug 29 21:20:21 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:21.834996 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1e54ef9-47c0-4e23-8f41-599af440193a None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:21.890469 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:21.960886 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70be9fac-dc03-46a2-a6d2-d07e2dd59ca5 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:21.963961 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1e54ef9-47c0-4e23-8f41-599af440193a None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:22.003630 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b732494e-4755-4553-a5cb-e19d5b84a0e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:22.004368 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b732494e-4755-4553-a5cb-e19d5b84a0e4 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:22.004649 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b732494e-4755-4553-a5cb-e19d5b84a0e4 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:22.004918 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b732494e-4755-4553-a5cb-e19d5b84a0e4 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:22.018726 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f235c2d-af86-490a-b49d-65fc0ddd26dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:22.019913 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f235c2d-af86-490a-b49d-65fc0ddd26dd None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:22.020237 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f235c2d-af86-490a-b49d-65fc0ddd26dd None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:22.020511 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f235c2d-af86-490a-b49d-65fc0ddd26dd None admin] PATH_INFO: `/v3/users/362070d0dbc049258a4ff352e5f937aa` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:22.024262 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f235c2d-af86-490a-b49d-65fc0ddd26dd None admin] RBAC: Authorizing `identity:get_user(user_id=362070d0dbc049258a4ff352e5f937aa)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:22.027642 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f235c2d-af86-490a-b49d-65fc0ddd26dd None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:22.030428 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b732494e-4755-4553-a5cb-e19d5b84a0e4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 21:20:22.033220 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 179/361] 10.222.0.90 () {56 vars in 1177 bytes} [Mon Aug 29 21:20:21 2022] GET /identity/v3/users/362070d0dbc049258a4ff352e5f937aa => generated 293 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:22.034238 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f235c2d-af86-490a-b49d-65fc0ddd26dd None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:22.042229 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-b732494e-4755-4553-a5cb-e19d5b84a0e4 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 29 21:20:22.042229 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:22.042229 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:22.042229 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:22.042229 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:22.042229 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:22.042229 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:22.042229 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:22.042229 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:22.042229 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:22.042229 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:22.042229 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:22.042229 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 21:20:22.042229 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 21:20:22.042229 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 21:20:22.042229 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 21:20:22.043268 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 21:20:22.043268 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 21:20:22.043268 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 21:20:22.043268 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 21:20:22.043268 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:22.043268 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:22.043268 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:22.043268 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:22.043268 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:22.043268 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:22.043268 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:22.043268 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:22.043268 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:22.043268 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:22.043268 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:22.043268 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:22.044087 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:22.044087 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:22.044087 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:22.044087 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:22.044087 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 21:20:22.044087 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 21:20:22.044087 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 21:20:22.044087 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 21:20:22.044087 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 21:20:22.044087 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 21:20:22.044087 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 29 21:20:22.044087 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:20:22.046114 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 183/362] 10.222.0.90 () {56 vars in 1114 bytes} [Mon Aug 29 21:20:21 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 219 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:20:22.048764 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b732494e-4755-4553-a5cb-e19d5b84a0e4 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:22.123824 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:22.240830 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f235c2d-af86-490a-b49d-65fc0ddd26dd None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:22.249675 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b732494e-4755-4553-a5cb-e19d5b84a0e4 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:22.263527 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:22.310618 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1eb7b64-8e6b-410c-a886-8da3710fea96 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:22.311237 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1eb7b64-8e6b-410c-a886-8da3710fea96 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:22.311568 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1eb7b64-8e6b-410c-a886-8da3710fea96 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:22.311845 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1eb7b64-8e6b-410c-a886-8da3710fea96 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:22.312584 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1eb7b64-8e6b-410c-a886-8da3710fea96 None admin] RBAC: Adding query filter params (role.id=414a14ad48a345f89fd1045f5d8fd16c, user.id=362070d0dbc049258a4ff352e5f937aa, scope.system=all) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:22.312889 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1eb7b64-8e6b-410c-a886-8da3710fea96 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:22.314356 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1eb7b64-8e6b-410c-a886-8da3710fea96 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:22.318541 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34433ff2-5f93-4a37-8202-bee6b6986138 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:22.319179 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34433ff2-5f93-4a37-8202-bee6b6986138 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:22.319463 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34433ff2-5f93-4a37-8202-bee6b6986138 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:22.319739 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34433ff2-5f93-4a37-8202-bee6b6986138 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:22.330423 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 184/363] 10.222.0.90 () {56 vars in 1308 bytes} [Mon Aug 29 21:20:22 2022] GET /identity/v3/role_assignments?user.id=362070d0dbc049258a4ff352e5f937aa&scope.system=all&role.id=414a14ad48a345f89fd1045f5d8fd16c => generated 510 bytes in 284 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:22.331523 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1eb7b64-8e6b-410c-a886-8da3710fea96 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:22.424159 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34433ff2-5f93-4a37-8202-bee6b6986138 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:22.425557 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34433ff2-5f93-4a37-8202-bee6b6986138 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:22.470746 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:22.484530 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1eb7b64-8e6b-410c-a886-8da3710fea96 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:22.540896 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 180/364] 10.222.0.90 () {58 vars in 1282 bytes} [Mon Aug 29 21:20:22 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 507 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 29 21:20:22.542382 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c79e37-00d1-477b-92db-bfd0379e3caa None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:22.542618 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c79e37-00d1-477b-92db-bfd0379e3caa None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:22.542906 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c79e37-00d1-477b-92db-bfd0379e3caa None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:22.544464 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 181/365] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:20:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:20:22.545098 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ea9d787-b57b-406f-a556-631e8e494466 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:22.545724 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ea9d787-b57b-406f-a556-631e8e494466 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:22.546089 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ea9d787-b57b-406f-a556-631e8e494466 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:22.546365 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ea9d787-b57b-406f-a556-631e8e494466 None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:22.547042 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ea9d787-b57b-406f-a556-631e8e494466 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:22.547400 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ea9d787-b57b-406f-a556-631e8e494466 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:22.548828 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ea9d787-b57b-406f-a556-631e8e494466 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:22.550543 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63178d5e-3787-4002-9d3d-8b568397679a None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:22.551080 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63178d5e-3787-4002-9d3d-8b568397679a None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:22.551580 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63178d5e-3787-4002-9d3d-8b568397679a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:22.562350 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 185/366] 10.222.0.90 () {56 vars in 1111 bytes} [Mon Aug 29 21:20:22 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 231 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:22.563313 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ea9d787-b57b-406f-a556-631e8e494466 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:22.576717 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-63178d5e-3787-4002-9d3d-8b568397679a None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:20:22.578387 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-63178d5e-3787-4002-9d3d-8b568397679a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:20:22.592350 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:22.671837 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 182/367] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:20:22 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 21:20:22.673054 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63178d5e-3787-4002-9d3d-8b568397679a None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:22.683590 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ea9d787-b57b-406f-a556-631e8e494466 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:22.743655 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ba5ae2b-970c-4553-8c8f-5af50b631840 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:22.744320 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ba5ae2b-970c-4553-8c8f-5af50b631840 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:22.744602 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ba5ae2b-970c-4553-8c8f-5af50b631840 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:22.744900 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ba5ae2b-970c-4553-8c8f-5af50b631840 None admin] PATH_INFO: `/v3/projects/f6d8dbb619904d8889100044bad1bef6` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:22.746253 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ba5ae2b-970c-4553-8c8f-5af50b631840 None admin] RBAC: Authorizing `identity:get_project(project_id=f6d8dbb619904d8889100044bad1bef6)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:22.747687 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ba5ae2b-970c-4553-8c8f-5af50b631840 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:22.753962 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 186/368] 10.222.0.90 () {56 vars in 1186 bytes} [Mon Aug 29 21:20:22 2022] GET /identity/v3/projects/f6d8dbb619904d8889100044bad1bef6 => generated 302 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:22.757166 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ba5ae2b-970c-4553-8c8f-5af50b631840 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:22.775035 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:22.813527 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63178d5e-3787-4002-9d3d-8b568397679a None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:22.863389 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f628e27b-bc65-4f9c-810f-ad3049e2be5d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:22.864185 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f628e27b-bc65-4f9c-810f-ad3049e2be5d None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:22.864397 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f628e27b-bc65-4f9c-810f-ad3049e2be5d None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:22.864630 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f628e27b-bc65-4f9c-810f-ad3049e2be5d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:22.914933 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:22.938800 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ba5ae2b-970c-4553-8c8f-5af50b631840 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:22.983983 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f628e27b-bc65-4f9c-810f-ad3049e2be5d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:22.985645 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f628e27b-bc65-4f9c-810f-ad3049e2be5d None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:23.005085 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b47adb11-d9a3-4e3e-92c3-b3a774d95b56 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:23.005724 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47adb11-d9a3-4e3e-92c3-b3a774d95b56 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:23.006022 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47adb11-d9a3-4e3e-92c3-b3a774d95b56 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:23.006291 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47adb11-d9a3-4e3e-92c3-b3a774d95b56 None admin] PATH_INFO: `/v3/groups/792e5c9f64ca4f019cee6fd3fe995744` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:23.007444 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b47adb11-d9a3-4e3e-92c3-b3a774d95b56 None admin] RBAC: Authorizing `identity:get_group(group_id=792e5c9f64ca4f019cee6fd3fe995744)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:23.008893 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b47adb11-d9a3-4e3e-92c3-b3a774d95b56 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:23.012971 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 187/369] 10.222.0.90 () {56 vars in 1180 bytes} [Mon Aug 29 21:20:22 2022] GET /identity/v3/groups/792e5c9f64ca4f019cee6fd3fe995744 => generated 226 bytes in 256 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:20:23.021021 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b47adb11-d9a3-4e3e-92c3-b3a774d95b56 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:23.102361 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:23.149179 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 183/370] 10.222.0.90 () {58 vars in 1282 bytes} [Mon Aug 29 21:20:22 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 476 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 29 21:20:23.150952 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f628e27b-bc65-4f9c-810f-ad3049e2be5d None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:23.183523 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b47adb11-d9a3-4e3e-92c3-b3a774d95b56 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:23.243509 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f090c72b-51db-4b8b-bd81-c68c5e720d9c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:23.244415 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f090c72b-51db-4b8b-bd81-c68c5e720d9c None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:23.244941 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f090c72b-51db-4b8b-bd81-c68c5e720d9c None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:23.245378 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f090c72b-51db-4b8b-bd81-c68c5e720d9c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:23.246293 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f090c72b-51db-4b8b-bd81-c68c5e720d9c None admin] RBAC: Adding query filter params (group.id=792e5c9f64ca4f019cee6fd3fe995744, role.id=02e242d5520740d4b77a5f4008c7eb57, scope.project.id=f6d8dbb619904d8889100044bad1bef6) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:23.246837 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f090c72b-51db-4b8b-bd81-c68c5e720d9c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:23.248519 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f090c72b-51db-4b8b-bd81-c68c5e720d9c None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:23.262016 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 188/371] 10.222.0.90 () {56 vars in 1376 bytes} [Mon Aug 29 21:20:23 2022] GET /identity/v3/role_assignments?group.id=792e5c9f64ca4f019cee6fd3fe995744&scope.project.id=f6d8dbb619904d8889100044bad1bef6&role.id=02e242d5520740d4b77a5f4008c7eb57 => generated 260 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:20:23.263856 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f090c72b-51db-4b8b-bd81-c68c5e720d9c None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:23.276510 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f628e27b-bc65-4f9c-810f-ad3049e2be5d None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:23.304246 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:23.326036 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdb305ef-41c3-413c-abf1-a0f83a72621b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:23.326740 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdb305ef-41c3-413c-abf1-a0f83a72621b None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:23.327020 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdb305ef-41c3-413c-abf1-a0f83a72621b None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:23.327285 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdb305ef-41c3-413c-abf1-a0f83a72621b None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:23.356520 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdb305ef-41c3-413c-abf1-a0f83a72621b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 21:20:23.363034 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-cdb305ef-41c3-413c-abf1-a0f83a72621b None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 21:20:23.363034 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:23.363034 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:23.363034 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:23.363034 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:23.363034 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:23.363034 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:23.363034 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:23.363034 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:23.363034 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:23.363034 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:23.363034 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:23.363034 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 21:20:23.363034 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 21:20:23.363034 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 21:20:23.363034 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 21:20:23.364527 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 21:20:23.364527 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 21:20:23.364527 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 21:20:23.364527 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 21:20:23.364527 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:23.364527 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:23.364527 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:23.364527 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:23.364527 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:23.364527 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:23.364527 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:23.364527 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:23.364527 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:23.364527 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:23.364527 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:23.364527 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:23.365332 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:23.365332 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:23.365332 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:23.365332 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:23.365332 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 21:20:23.365332 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 21:20:23.365332 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 21:20:23.365332 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 21:20:23.365332 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 21:20:23.365332 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 21:20:23.365332 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 21:20:23.365332 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:20:23.371896 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 184/372] 10.222.0.90 () {56 vars in 1096 bytes} [Mon Aug 29 21:20:23 2022] GET /identity/v3/roles/admin => generated 83 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:20:23.378749 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdb305ef-41c3-413c-abf1-a0f83a72621b None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:23.437444 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f090c72b-51db-4b8b-bd81-c68c5e720d9c None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:23.456465 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:23.489701 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-219e58ba-fe65-4ffc-9034-347e89b28536 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:23.490421 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-219e58ba-fe65-4ffc-9034-347e89b28536 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:23.490703 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-219e58ba-fe65-4ffc-9034-347e89b28536 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:23.490967 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-219e58ba-fe65-4ffc-9034-347e89b28536 None admin] PATH_INFO: `/v3/projects/f6d8dbb619904d8889100044bad1bef6` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:23.494718 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-219e58ba-fe65-4ffc-9034-347e89b28536 None admin] RBAC: Authorizing `identity:get_project(project_id=f6d8dbb619904d8889100044bad1bef6)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:23.496208 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-219e58ba-fe65-4ffc-9034-347e89b28536 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:23.499511 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 189/373] 10.222.0.90 () {56 vars in 1186 bytes} [Mon Aug 29 21:20:23 2022] GET /identity/v3/projects/f6d8dbb619904d8889100044bad1bef6 => generated 302 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:23.513385 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-219e58ba-fe65-4ffc-9034-347e89b28536 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:23.585488 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdb305ef-41c3-413c-abf1-a0f83a72621b None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:23.628732 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89723120-6aa5-4122-ad9e-9757267fc325 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:23.630131 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89723120-6aa5-4122-ad9e-9757267fc325 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:23.630581 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89723120-6aa5-4122-ad9e-9757267fc325 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:23.631012 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89723120-6aa5-4122-ad9e-9757267fc325 None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:23.631921 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89723120-6aa5-4122-ad9e-9757267fc325 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:23.632428 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89723120-6aa5-4122-ad9e-9757267fc325 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:23.634052 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89723120-6aa5-4122-ad9e-9757267fc325 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:23.643866 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 185/374] 10.222.0.90 () {56 vars in 1099 bytes} [Mon Aug 29 21:20:23 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 265 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:23.652927 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89723120-6aa5-4122-ad9e-9757267fc325 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:23.682645 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:23.704273 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-219e58ba-fe65-4ffc-9034-347e89b28536 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:23.749119 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c60f4776-ffee-48fe-bb0d-20290939de6e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:23.749673 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c60f4776-ffee-48fe-bb0d-20290939de6e None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:23.750074 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c60f4776-ffee-48fe-bb0d-20290939de6e None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:23.750346 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c60f4776-ffee-48fe-bb0d-20290939de6e None admin] PATH_INFO: `/v3/groups/792e5c9f64ca4f019cee6fd3fe995744` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:23.751564 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c60f4776-ffee-48fe-bb0d-20290939de6e None admin] RBAC: Authorizing `identity:get_group(group_id=792e5c9f64ca4f019cee6fd3fe995744)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:23.753281 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c60f4776-ffee-48fe-bb0d-20290939de6e None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:23.756131 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 190/375] 10.222.0.90 () {56 vars in 1180 bytes} [Mon Aug 29 21:20:23 2022] GET /identity/v3/groups/792e5c9f64ca4f019cee6fd3fe995744 => generated 226 bytes in 243 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:20:23.762713 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c60f4776-ffee-48fe-bb0d-20290939de6e None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:23.800177 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:23.815927 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89723120-6aa5-4122-ad9e-9757267fc325 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:23.881154 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-023d5899-3946-4915-8220-fcc4ebbce246 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:23.882682 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-023d5899-3946-4915-8220-fcc4ebbce246 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:23.883306 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-023d5899-3946-4915-8220-fcc4ebbce246 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:23.885685 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-023d5899-3946-4915-8220-fcc4ebbce246 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:23.948409 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c60f4776-ffee-48fe-bb0d-20290939de6e None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:23.993208 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2697f16-7836-42b0-b33f-f8282812c55e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:23.994229 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2697f16-7836-42b0-b33f-f8282812c55e None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:23.994724 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2697f16-7836-42b0-b33f-f8282812c55e None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:23.995752 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2697f16-7836-42b0-b33f-f8282812c55e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:23.996982 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2697f16-7836-42b0-b33f-f8282812c55e None admin] RBAC: Adding query filter params (group.id=792e5c9f64ca4f019cee6fd3fe995744, role.id=9745586777bf473bab7d887bf25bf236, scope.project.id=f6d8dbb619904d8889100044bad1bef6) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:23.997513 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2697f16-7836-42b0-b33f-f8282812c55e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:23.997627 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-023d5899-3946-4915-8220-fcc4ebbce246 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:23.999022 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-023d5899-3946-4915-8220-fcc4ebbce246 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:23.999607 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2697f16-7836-42b0-b33f-f8282812c55e None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:24.008711 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 191/376] 10.222.0.90 () {56 vars in 1376 bytes} [Mon Aug 29 21:20:23 2022] GET /identity/v3/role_assignments?group.id=792e5c9f64ca4f019cee6fd3fe995744&scope.project.id=f6d8dbb619904d8889100044bad1bef6&role.id=9745586777bf473bab7d887bf25bf236 => generated 260 bytes in 246 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:20:24.010879 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7eab0c4-5b56-4cfa-ad04-eadc57d448eb None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:24.010879 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7eab0c4-5b56-4cfa-ad04-eadc57d448eb None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:24.010879 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7eab0c4-5b56-4cfa-ad04-eadc57d448eb None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:24.012187 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 192/377] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:20:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:20:24.018640 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ab7b039-f661-468b-b743-bea082c19a11 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:24.018952 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ab7b039-f661-468b-b743-bea082c19a11 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:24.020470 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ab7b039-f661-468b-b743-bea082c19a11 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:24.058810 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-2ab7b039-f661-468b-b743-bea082c19a11 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:20:24.060041 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-2ab7b039-f661-468b-b743-bea082c19a11 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:20:24.087973 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:24.155327 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 193/378] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:20:24 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 21:20:24.180107 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2443597e-d1be-4407-b698-45e8c3ee3e8a None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:24.180761 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2443597e-d1be-4407-b698-45e8c3ee3e8a None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:24.180926 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2443597e-d1be-4407-b698-45e8c3ee3e8a None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:24.182589 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 194/379] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:20:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 29 21:20:24.187739 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2ac0c00-fae6-4dd6-a793-f98eb0934872 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:24.188025 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2ac0c00-fae6-4dd6-a793-f98eb0934872 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:24.188306 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2ac0c00-fae6-4dd6-a793-f98eb0934872 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:24.201868 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:24.209917 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-e2ac0c00-fae6-4dd6-a793-f98eb0934872 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:20:24.211140 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-e2ac0c00-fae6-4dd6-a793-f98eb0934872 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:20:24.222239 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 186/380] 10.222.0.90 () {58 vars in 1282 bytes} [Mon Aug 29 21:20:23 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 570 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 29 21:20:24.228093 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-023d5899-3946-4915-8220-fcc4ebbce246 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:24.344657 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 195/381] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:20:24 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 29 21:20:24.348142 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-023d5899-3946-4915-8220-fcc4ebbce246 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:24.386842 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38d4a368-5e90-4b67-b2b4-8a40106c0408 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:24.387749 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38d4a368-5e90-4b67-b2b4-8a40106c0408 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:24.388199 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38d4a368-5e90-4b67-b2b4-8a40106c0408 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:24.388628 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38d4a368-5e90-4b67-b2b4-8a40106c0408 None admin] PATH_INFO: `/v3/projects/fa1ebaa0d49241599f614eb2efe4d8c8` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:24.390663 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38d4a368-5e90-4b67-b2b4-8a40106c0408 None admin] RBAC: Authorizing `identity:get_project(project_id=fa1ebaa0d49241599f614eb2efe4d8c8)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:24.392270 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38d4a368-5e90-4b67-b2b4-8a40106c0408 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:24.394949 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 187/382] 10.222.0.90 () {56 vars in 1186 bytes} [Mon Aug 29 21:20:24 2022] GET /identity/v3/projects/fa1ebaa0d49241599f614eb2efe4d8c8 => generated 344 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:20:24.400927 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2ac0c00-fae6-4dd6-a793-f98eb0934872 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:24.511392 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:24.512677 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38d4a368-5e90-4b67-b2b4-8a40106c0408 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:24.558968 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2ac0c00-fae6-4dd6-a793-f98eb0934872 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:24.626247 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:24.631461 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bf1a37e-cd2f-4ef3-945a-5212d9d99886 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:24.632115 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bf1a37e-cd2f-4ef3-945a-5212d9d99886 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:24.632402 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bf1a37e-cd2f-4ef3-945a-5212d9d99886 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:24.632680 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bf1a37e-cd2f-4ef3-945a-5212d9d99886 None admin] PATH_INFO: `/v3/groups/1d230e348ec544faac7eeff5f0a65f0a` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:24.646273 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bf1a37e-cd2f-4ef3-945a-5212d9d99886 None admin] RBAC: Authorizing `identity:get_group(group_id=1d230e348ec544faac7eeff5f0a65f0a)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:24.647761 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bf1a37e-cd2f-4ef3-945a-5212d9d99886 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:24.649450 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 196/383] 10.222.0.90 () {56 vars in 1180 bytes} [Mon Aug 29 21:20:24 2022] GET /identity/v3/groups/1d230e348ec544faac7eeff5f0a65f0a => generated 229 bytes in 249 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:24.658242 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bf1a37e-cd2f-4ef3-945a-5212d9d99886 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:24.711911 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38d4a368-5e90-4b67-b2b4-8a40106c0408 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:24.779987 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a1fbd8c-d622-4535-8e6a-57422e055066 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:24.780649 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a1fbd8c-d622-4535-8e6a-57422e055066 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:24.780936 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a1fbd8c-d622-4535-8e6a-57422e055066 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:24.781232 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a1fbd8c-d622-4535-8e6a-57422e055066 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:24.801323 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a1fbd8c-d622-4535-8e6a-57422e055066 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 21:20:24.809855 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-5a1fbd8c-d622-4535-8e6a-57422e055066 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 29 21:20:24.809855 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:24.809855 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:24.809855 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:24.809855 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:24.809855 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:24.809855 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:24.809855 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:24.809855 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:24.809855 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:24.809855 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:24.809855 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:24.809855 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 21:20:24.809855 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 21:20:24.809855 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 21:20:24.809855 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 21:20:24.810860 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 21:20:24.810860 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 21:20:24.810860 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 21:20:24.810860 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 21:20:24.810860 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:24.810860 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:24.810860 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:24.810860 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:24.810860 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:24.810860 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:24.810860 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:24.810860 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:24.810860 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:24.810860 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:24.810860 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:24.810860 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:24.811714 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:24.811714 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:24.811714 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:24.811714 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:24.811714 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 21:20:24.811714 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 21:20:24.811714 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 21:20:24.811714 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 21:20:24.811714 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 21:20:24.811714 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 21:20:24.811714 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 29 21:20:24.811714 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:20:24.812858 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 188/384] 10.222.0.90 () {56 vars in 1099 bytes} [Mon Aug 29 21:20:24 2022] GET /identity/v3/roles/member => generated 84 bytes in 299 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:20:24.813104 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a1fbd8c-d622-4535-8e6a-57422e055066 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:24.827318 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bf1a37e-cd2f-4ef3-945a-5212d9d99886 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:24.894959 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-448a65a1-7fb9-49c0-a8c7-dd620ffe4455 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:24.895933 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-448a65a1-7fb9-49c0-a8c7-dd620ffe4455 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:24.896375 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-448a65a1-7fb9-49c0-a8c7-dd620ffe4455 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:24.896809 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-448a65a1-7fb9-49c0-a8c7-dd620ffe4455 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:24.898201 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-448a65a1-7fb9-49c0-a8c7-dd620ffe4455 None admin] RBAC: Adding query filter params (group.id=1d230e348ec544faac7eeff5f0a65f0a, role.id=a6e562cdc96c4be691d638990f8403fd, scope.project.id=fa1ebaa0d49241599f614eb2efe4d8c8) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:24.898683 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-448a65a1-7fb9-49c0-a8c7-dd620ffe4455 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:24.900818 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-448a65a1-7fb9-49c0-a8c7-dd620ffe4455 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:24.909307 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 197/385] 10.222.0.90 () {56 vars in 1376 bytes} [Mon Aug 29 21:20:24 2022] GET /identity/v3/role_assignments?group.id=1d230e348ec544faac7eeff5f0a65f0a&scope.project.id=fa1ebaa0d49241599f614eb2efe4d8c8&role.id=a6e562cdc96c4be691d638990f8403fd => generated 260 bytes in 251 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:24.910966 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-448a65a1-7fb9-49c0-a8c7-dd620ffe4455 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:24.940013 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:24.972075 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a1fbd8c-d622-4535-8e6a-57422e055066 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:25.000304 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:25.018405 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fdf1a0c-9d3b-4b0d-8efa-c21bf7b6d399 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:25.019059 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fdf1a0c-9d3b-4b0d-8efa-c21bf7b6d399 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:25.020086 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fdf1a0c-9d3b-4b0d-8efa-c21bf7b6d399 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:25.020238 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fdf1a0c-9d3b-4b0d-8efa-c21bf7b6d399 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:25.034439 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fdf1a0c-9d3b-4b0d-8efa-c21bf7b6d399 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 21:20:25.041351 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-3fdf1a0c-9d3b-4b0d-8efa-c21bf7b6d399 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 29 21:20:25.041351 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:25.041351 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:25.041351 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:25.041351 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:25.041351 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:25.041351 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:25.041351 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:25.041351 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:25.041351 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:25.041351 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:25.041351 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:25.041351 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 21:20:25.041351 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 21:20:25.041351 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 21:20:25.041351 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 21:20:25.042359 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 21:20:25.042359 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 21:20:25.042359 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 21:20:25.042359 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 21:20:25.042359 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:25.042359 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:25.042359 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:25.042359 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:25.042359 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:25.042359 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:25.042359 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:25.042359 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:25.042359 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:25.042359 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:25.042359 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:25.042359 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:25.043202 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:25.043202 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:25.043202 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:25.043202 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:25.043202 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 21:20:25.043202 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 21:20:25.043202 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 21:20:25.043202 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 21:20:25.043202 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 21:20:25.043202 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 21:20:25.043202 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 29 21:20:25.043202 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:20:25.044318 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 189/386] 10.222.0.90 () {56 vars in 1114 bytes} [Mon Aug 29 21:20:24 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 231 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:20:25.050252 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fdf1a0c-9d3b-4b0d-8efa-c21bf7b6d399 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:25.194837 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-448a65a1-7fb9-49c0-a8c7-dd620ffe4455 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:25.201940 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fdf1a0c-9d3b-4b0d-8efa-c21bf7b6d399 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:25.269072 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32103cda-a57b-487f-975a-9391a0cf592a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:25.270854 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32103cda-a57b-487f-975a-9391a0cf592a None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:25.271626 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32103cda-a57b-487f-975a-9391a0cf592a None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:25.272520 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32103cda-a57b-487f-975a-9391a0cf592a None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:25.273819 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32103cda-a57b-487f-975a-9391a0cf592a None admin] RBAC: Adding query filter params (name=member) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:25.274671 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32103cda-a57b-487f-975a-9391a0cf592a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:25.276476 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32103cda-a57b-487f-975a-9391a0cf592a None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:25.279012 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-805efd45-6685-4ad2-b577-52326fcd0cb7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:25.279787 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-805efd45-6685-4ad2-b577-52326fcd0cb7 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:25.280227 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-805efd45-6685-4ad2-b577-52326fcd0cb7 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:25.280642 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-805efd45-6685-4ad2-b577-52326fcd0cb7 None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:25.281444 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-805efd45-6685-4ad2-b577-52326fcd0cb7 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:25.281910 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-805efd45-6685-4ad2-b577-52326fcd0cb7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:25.283663 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-805efd45-6685-4ad2-b577-52326fcd0cb7 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:25.297144 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 198/387] 10.222.0.90 () {56 vars in 1101 bytes} [Mon Aug 29 21:20:24 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 387 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:25.301373 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 190/388] 10.222.0.90 () {56 vars in 1111 bytes} [Mon Aug 29 21:20:25 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 251 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:20:25.306802 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32103cda-a57b-487f-975a-9391a0cf592a None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:25.308267 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-805efd45-6685-4ad2-b577-52326fcd0cb7 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:25.362609 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:25.431303 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-805efd45-6685-4ad2-b577-52326fcd0cb7 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:25.468619 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a599fc1-9031-42f0-9e8d-339ab5aa6e24 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:25.469114 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:25.470978 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a599fc1-9031-42f0-9e8d-339ab5aa6e24 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:25.472067 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a599fc1-9031-42f0-9e8d-339ab5aa6e24 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:25.472067 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a599fc1-9031-42f0-9e8d-339ab5aa6e24 None admin] PATH_INFO: `/v3/groups/792e5c9f64ca4f019cee6fd3fe995744` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:25.474403 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a599fc1-9031-42f0-9e8d-339ab5aa6e24 None admin] RBAC: Authorizing `identity:get_group(group_id=792e5c9f64ca4f019cee6fd3fe995744)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:25.475996 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a599fc1-9031-42f0-9e8d-339ab5aa6e24 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:25.479647 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 191/389] 10.222.0.90 () {56 vars in 1180 bytes} [Mon Aug 29 21:20:25 2022] GET /identity/v3/groups/792e5c9f64ca4f019cee6fd3fe995744 => generated 226 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:25.485240 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a599fc1-9031-42f0-9e8d-339ab5aa6e24 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:25.537636 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32103cda-a57b-487f-975a-9391a0cf592a None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:25.599082 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a599fc1-9031-42f0-9e8d-339ab5aa6e24 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:25.608350 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7091d11b-4b7e-4da1-86b0-3722611809d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:25.609072 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7091d11b-4b7e-4da1-86b0-3722611809d6 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:25.609372 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7091d11b-4b7e-4da1-86b0-3722611809d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:25.609528 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7091d11b-4b7e-4da1-86b0-3722611809d6 None admin] PATH_INFO: `/v3/groups/792e5c9f64ca4f019cee6fd3fe995744` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:25.611023 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:25.612787 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7091d11b-4b7e-4da1-86b0-3722611809d6 None admin] RBAC: Authorizing `identity:get_group(group_id=792e5c9f64ca4f019cee6fd3fe995744)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:25.614543 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7091d11b-4b7e-4da1-86b0-3722611809d6 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:25.616526 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 199/390] 10.222.0.90 () {56 vars in 1180 bytes} [Mon Aug 29 21:20:25 2022] GET /identity/v3/groups/792e5c9f64ca4f019cee6fd3fe995744 => generated 226 bytes in 310 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:25.622002 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7091d11b-4b7e-4da1-86b0-3722611809d6 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:25.638639 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9de8a73a-9d17-4f6a-b0da-abb9f7a8b867 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:25.639538 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9de8a73a-9d17-4f6a-b0da-abb9f7a8b867 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:25.639977 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9de8a73a-9d17-4f6a-b0da-abb9f7a8b867 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:25.640577 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9de8a73a-9d17-4f6a-b0da-abb9f7a8b867 None admin] PATH_INFO: `/v3/projects/381cc34791e147329798cd454f99f622` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:25.646820 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9de8a73a-9d17-4f6a-b0da-abb9f7a8b867 None admin] RBAC: Authorizing `identity:get_project(project_id=381cc34791e147329798cd454f99f622)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:25.649356 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9de8a73a-9d17-4f6a-b0da-abb9f7a8b867 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:25.652037 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 192/391] 10.222.0.90 () {56 vars in 1186 bytes} [Mon Aug 29 21:20:25 2022] GET /identity/v3/projects/381cc34791e147329798cd454f99f622 => generated 298 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:25.664291 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9de8a73a-9d17-4f6a-b0da-abb9f7a8b867 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:25.821361 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7091d11b-4b7e-4da1-86b0-3722611809d6 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:25.839367 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9de8a73a-9d17-4f6a-b0da-abb9f7a8b867 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:25.867321 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a02e6cb-975a-4d2b-b3a3-684d2d4441da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:25.867970 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a02e6cb-975a-4d2b-b3a3-684d2d4441da None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:25.869047 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a02e6cb-975a-4d2b-b3a3-684d2d4441da None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:25.869047 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a02e6cb-975a-4d2b-b3a3-684d2d4441da None admin] PATH_INFO: `/v3/projects/381cc34791e147329798cd454f99f622` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:25.872441 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a02e6cb-975a-4d2b-b3a3-684d2d4441da None admin] RBAC: Authorizing `identity:get_project(project_id=381cc34791e147329798cd454f99f622)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:25.873923 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a02e6cb-975a-4d2b-b3a3-684d2d4441da None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:25.875735 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 200/392] 10.222.0.90 () {56 vars in 1186 bytes} [Mon Aug 29 21:20:25 2022] GET /identity/v3/projects/381cc34791e147329798cd454f99f622 => generated 298 bytes in 254 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:20:25.883722 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c52bd2e-0310-4169-b5eb-388b7fa8d1e8 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:25.883722 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c52bd2e-0310-4169-b5eb-388b7fa8d1e8 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:25.883722 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c52bd2e-0310-4169-b5eb-388b7fa8d1e8 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:25.884176 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 201/393] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:20:25 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:20:25.885331 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c52bd2e-0310-4169-b5eb-388b7fa8d1e8 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:25.896045 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c86d466c-190d-41f8-b210-cba50d1ef51d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:25.899126 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c86d466c-190d-41f8-b210-cba50d1ef51d None admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:25.899675 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c86d466c-190d-41f8-b210-cba50d1ef51d None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:25.900156 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c86d466c-190d-41f8-b210-cba50d1ef51d None admin] PATH_INFO: `/v3/projects/381cc34791e147329798cd454f99f622/groups/792e5c9f64ca4f019cee6fd3fe995744/roles/9745586777bf473bab7d887bf25bf236` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:25.902157 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:25.902946 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c86d466c-190d-41f8-b210-cba50d1ef51d None admin] RBAC: Authorizing `identity:create_grant(project_id=381cc34791e147329798cd454f99f622, group_id=792e5c9f64ca4f019cee6fd3fe995744, role_id=9745586777bf473bab7d887bf25bf236)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:25.904639 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c86d466c-190d-41f8-b210-cba50d1ef51d None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:25.926361 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 193/394] 10.222.0.90 () {58 vars in 1442 bytes} [Mon Aug 29 21:20:25 2022] PUT /identity/v3/projects/381cc34791e147329798cd454f99f622/groups/792e5c9f64ca4f019cee6fd3fe995744/roles/9745586777bf473bab7d887bf25bf236 => generated 0 bytes in 262 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 21:20:25.928225 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56d5752d-bee6-457a-81f1-9eeb306df83d None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:25.928757 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56d5752d-bee6-457a-81f1-9eeb306df83d None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:25.929360 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56d5752d-bee6-457a-81f1-9eeb306df83d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:25.951958 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-56d5752d-bee6-457a-81f1-9eeb306df83d None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:20:25.953817 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-56d5752d-bee6-457a-81f1-9eeb306df83d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:20:25.994761 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:26.004376 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c52bd2e-0310-4169-b5eb-388b7fa8d1e8 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:26.044798 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e8a67b4-75b3-470a-be8c-1d79eaa0a39c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:26.045519 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e8a67b4-75b3-470a-be8c-1d79eaa0a39c None admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:26.045763 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e8a67b4-75b3-470a-be8c-1d79eaa0a39c None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:26.046103 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e8a67b4-75b3-470a-be8c-1d79eaa0a39c None admin] PATH_INFO: `/v3/projects/381cc34791e147329798cd454f99f622/groups/792e5c9f64ca4f019cee6fd3fe995744/roles/02e242d5520740d4b77a5f4008c7eb57` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:26.048437 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 194/395] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:20:25 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 21:20:26.049539 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e8a67b4-75b3-470a-be8c-1d79eaa0a39c None admin] RBAC: Authorizing `identity:create_grant(project_id=381cc34791e147329798cd454f99f622, group_id=792e5c9f64ca4f019cee6fd3fe995744, role_id=02e242d5520740d4b77a5f4008c7eb57)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:26.053950 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e8a67b4-75b3-470a-be8c-1d79eaa0a39c None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:26.074235 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 202/396] 10.222.0.90 () {58 vars in 1442 bytes} [Mon Aug 29 21:20:25 2022] PUT /identity/v3/projects/381cc34791e147329798cd454f99f622/groups/792e5c9f64ca4f019cee6fd3fe995744/roles/02e242d5520740d4b77a5f4008c7eb57 => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 21:20:26.348073 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56d5752d-bee6-457a-81f1-9eeb306df83d None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:26.484611 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56d5752d-bee6-457a-81f1-9eeb306df83d None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:26.529725 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0fcd312-a918-419e-96e2-78b15d6e3dc2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:26.530393 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0fcd312-a918-419e-96e2-78b15d6e3dc2 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:26.530670 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0fcd312-a918-419e-96e2-78b15d6e3dc2 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:26.530939 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0fcd312-a918-419e-96e2-78b15d6e3dc2 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:26.545254 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0fcd312-a918-419e-96e2-78b15d6e3dc2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 21:20:26.550289 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-b0fcd312-a918-419e-96e2-78b15d6e3dc2 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 29 21:20:26.550289 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:26.550289 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:26.550289 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:26.550289 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:26.550289 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:26.550289 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:26.550289 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:26.550289 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:26.550289 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:26.550289 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:26.550289 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:26.550289 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 21:20:26.550289 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 21:20:26.550289 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 21:20:26.550289 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 21:20:26.551502 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 21:20:26.551502 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 21:20:26.551502 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 21:20:26.551502 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 21:20:26.551502 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:26.551502 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:26.551502 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:26.551502 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:26.551502 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:26.551502 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:26.551502 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:26.551502 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:26.551502 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:26.551502 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:26.551502 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:26.551502 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:26.552324 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:26.552324 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:26.552324 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:26.552324 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:26.552324 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 21:20:26.552324 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 21:20:26.552324 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 21:20:26.552324 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 21:20:26.552324 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 21:20:26.552324 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 21:20:26.552324 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 29 21:20:26.552324 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:20:26.552324 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 195/397] 10.222.0.90 () {56 vars in 1099 bytes} [Mon Aug 29 21:20:26 2022] GET /identity/v3/roles/member => generated 84 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:20:26.562903 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e8a67b4-75b3-470a-be8c-1d79eaa0a39c None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:26.656954 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:26.705276 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e8a67b4-75b3-470a-be8c-1d79eaa0a39c None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:26.744750 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d10b35f3-408c-4178-a85b-717fa3352610 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:26.745071 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d10b35f3-408c-4178-a85b-717fa3352610 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:26.751458 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d10b35f3-408c-4178-a85b-717fa3352610 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:26.752872 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 196/398] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:20:26 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 29 21:20:26.753380 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38e7d636-68f7-4815-9d54-ce9632983b99 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:26.755111 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38e7d636-68f7-4815-9d54-ce9632983b99 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:26.755752 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38e7d636-68f7-4815-9d54-ce9632983b99 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:26.756255 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38e7d636-68f7-4815-9d54-ce9632983b99 None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:26.757224 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38e7d636-68f7-4815-9d54-ce9632983b99 None admin] RBAC: Adding query filter params (name=member) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:26.757753 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38e7d636-68f7-4815-9d54-ce9632983b99 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:26.759390 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38e7d636-68f7-4815-9d54-ce9632983b99 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:26.774150 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 203/399] 10.222.0.90 () {56 vars in 1101 bytes} [Mon Aug 29 21:20:26 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:26.775259 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c26cf7b9-8858-4772-b72b-d6d055b16481 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:26.775543 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c26cf7b9-8858-4772-b72b-d6d055b16481 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:26.775814 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c26cf7b9-8858-4772-b72b-d6d055b16481 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:26.779374 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38e7d636-68f7-4815-9d54-ce9632983b99 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:26.818881 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-c26cf7b9-8858-4772-b72b-d6d055b16481 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:20:26.820092 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-c26cf7b9-8858-4772-b72b-d6d055b16481 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:20:26.865847 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:26.906795 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38e7d636-68f7-4815-9d54-ce9632983b99 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:26.943918 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:26.951946 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c153b048-41e3-4167-8c9b-c5d979008b40 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:26.952597 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c153b048-41e3-4167-8c9b-c5d979008b40 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:26.952882 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c153b048-41e3-4167-8c9b-c5d979008b40 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:26.953256 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c153b048-41e3-4167-8c9b-c5d979008b40 None admin] PATH_INFO: `/v3/groups/792e5c9f64ca4f019cee6fd3fe995744` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:26.954417 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c153b048-41e3-4167-8c9b-c5d979008b40 None admin] RBAC: Authorizing `identity:get_group(group_id=792e5c9f64ca4f019cee6fd3fe995744)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:26.955452 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 197/400] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:20:26 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 21:20:26.955959 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c153b048-41e3-4167-8c9b-c5d979008b40 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:26.962712 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 204/401] 10.222.0.90 () {56 vars in 1180 bytes} [Mon Aug 29 21:20:26 2022] GET /identity/v3/groups/792e5c9f64ca4f019cee6fd3fe995744 => generated 226 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:26.968147 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c26cf7b9-8858-4772-b72b-d6d055b16481 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:27.124572 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c26cf7b9-8858-4772-b72b-d6d055b16481 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:27.171541 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-188595db-e448-44d3-89c6-4ac35e3ff7e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:27.172183 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-188595db-e448-44d3-89c6-4ac35e3ff7e0 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:27.172466 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-188595db-e448-44d3-89c6-4ac35e3ff7e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:27.172744 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-188595db-e448-44d3-89c6-4ac35e3ff7e0 None admin] PATH_INFO: `/v3/projects/f6d8dbb619904d8889100044bad1bef6` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:27.173899 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-188595db-e448-44d3-89c6-4ac35e3ff7e0 None admin] RBAC: Authorizing `identity:get_project(project_id=f6d8dbb619904d8889100044bad1bef6)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:27.175334 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-188595db-e448-44d3-89c6-4ac35e3ff7e0 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:27.177344 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 198/402] 10.222.0.90 () {56 vars in 1186 bytes} [Mon Aug 29 21:20:26 2022] GET /identity/v3/projects/f6d8dbb619904d8889100044bad1bef6 => generated 302 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:27.183162 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c153b048-41e3-4167-8c9b-c5d979008b40 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:27.316878 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c153b048-41e3-4167-8c9b-c5d979008b40 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:27.349189 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-188595db-e448-44d3-89c6-4ac35e3ff7e0 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:27.354711 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c475853-3b0d-4f2c-be83-ef345729ff43 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:27.355401 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c475853-3b0d-4f2c-be83-ef345729ff43 None admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:27.355687 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c475853-3b0d-4f2c-be83-ef345729ff43 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:27.355963 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c475853-3b0d-4f2c-be83-ef345729ff43 None admin] PATH_INFO: `/v3/projects/f6d8dbb619904d8889100044bad1bef6/groups/792e5c9f64ca4f019cee6fd3fe995744/roles/02e242d5520740d4b77a5f4008c7eb57` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:27.357933 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c475853-3b0d-4f2c-be83-ef345729ff43 None admin] RBAC: Authorizing `identity:create_grant(project_id=f6d8dbb619904d8889100044bad1bef6, group_id=792e5c9f64ca4f019cee6fd3fe995744, role_id=02e242d5520740d4b77a5f4008c7eb57)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:27.359392 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c475853-3b0d-4f2c-be83-ef345729ff43 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:27.381151 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 205/403] 10.222.0.90 () {58 vars in 1442 bytes} [Mon Aug 29 21:20:27 2022] PUT /identity/v3/projects/f6d8dbb619904d8889100044bad1bef6/groups/792e5c9f64ca4f019cee6fd3fe995744/roles/02e242d5520740d4b77a5f4008c7eb57 => generated 0 bytes in 198 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:20:27.397152 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:27.477995 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-188595db-e448-44d3-89c6-4ac35e3ff7e0 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:27.514918 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22095e61-ffd0-476e-977f-2da7304910af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:27.515581 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22095e61-ffd0-476e-977f-2da7304910af None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:27.515864 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22095e61-ffd0-476e-977f-2da7304910af None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:27.516136 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22095e61-ffd0-476e-977f-2da7304910af None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:27.527722 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22095e61-ffd0-476e-977f-2da7304910af None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 21:20:27.533180 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-22095e61-ffd0-476e-977f-2da7304910af None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 29 21:20:27.533180 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:27.533180 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:27.533180 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:27.533180 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:27.533180 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:27.533180 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:27.533180 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:27.533180 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:27.533180 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:27.533180 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:27.533180 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:27.533180 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 21:20:27.533180 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 21:20:27.533180 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 21:20:27.533180 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 21:20:27.534268 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 21:20:27.534268 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 21:20:27.534268 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 21:20:27.534268 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 21:20:27.534268 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:27.534268 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:27.534268 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:27.534268 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:27.534268 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:27.534268 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:27.534268 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:27.534268 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:27.534268 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:27.534268 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:27.534268 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:27.534268 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:27.535849 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:27.535849 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:27.535849 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:27.535849 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:27.535849 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 21:20:27.535849 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 21:20:27.535849 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 21:20:27.535849 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 21:20:27.535849 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 21:20:27.535849 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 21:20:27.535849 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 29 21:20:27.535849 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:20:27.535849 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 199/404] 10.222.0.90 () {56 vars in 1114 bytes} [Mon Aug 29 21:20:27 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:20:27.541496 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c475853-3b0d-4f2c-be83-ef345729ff43 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:27.611899 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:27.640482 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c01334eb-edaf-4483-b245-ad20f45dfb00 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:27.640810 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c01334eb-edaf-4483-b245-ad20f45dfb00 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:27.642568 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c01334eb-edaf-4483-b245-ad20f45dfb00 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:27.642568 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 200/405] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:20:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 29 21:20:27.647773 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c80d0ab-48a7-4cf1-a437-1af4bd016023 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:27.648062 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c80d0ab-48a7-4cf1-a437-1af4bd016023 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:27.648342 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c80d0ab-48a7-4cf1-a437-1af4bd016023 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:27.668733 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-1c80d0ab-48a7-4cf1-a437-1af4bd016023 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:20:27.669210 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c475853-3b0d-4f2c-be83-ef345729ff43 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:27.669986 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-1c80d0ab-48a7-4cf1-a437-1af4bd016023 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:20:27.713435 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d576586-9fa5-4966-ac2b-2fe120b739ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:27.714092 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d576586-9fa5-4966-ac2b-2fe120b739ec None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:27.714367 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d576586-9fa5-4966-ac2b-2fe120b739ec None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:27.714644 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d576586-9fa5-4966-ac2b-2fe120b739ec None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:27.715600 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d576586-9fa5-4966-ac2b-2fe120b739ec None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:27.715905 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d576586-9fa5-4966-ac2b-2fe120b739ec None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:27.717373 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d576586-9fa5-4966-ac2b-2fe120b739ec None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:27.727722 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 206/406] 10.222.0.90 () {56 vars in 1111 bytes} [Mon Aug 29 21:20:27 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:27.733218 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d576586-9fa5-4966-ac2b-2fe120b739ec None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:27.751688 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 201/407] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:20:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 21:20:27.847914 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d576586-9fa5-4966-ac2b-2fe120b739ec None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:27.870833 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:27.892531 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f806c756-088e-4f69-abcc-ccea12835bfe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:27.893223 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f806c756-088e-4f69-abcc-ccea12835bfe None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:27.893510 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f806c756-088e-4f69-abcc-ccea12835bfe None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:27.893810 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f806c756-088e-4f69-abcc-ccea12835bfe None admin] PATH_INFO: `/v3/groups/792e5c9f64ca4f019cee6fd3fe995744` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:27.895646 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f806c756-088e-4f69-abcc-ccea12835bfe None admin] RBAC: Authorizing `identity:get_group(group_id=792e5c9f64ca4f019cee6fd3fe995744)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:27.897266 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f806c756-088e-4f69-abcc-ccea12835bfe None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:27.902840 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 207/408] 10.222.0.90 () {56 vars in 1180 bytes} [Mon Aug 29 21:20:27 2022] GET /identity/v3/groups/792e5c9f64ca4f019cee6fd3fe995744 => generated 226 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:27.908145 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c80d0ab-48a7-4cf1-a437-1af4bd016023 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:27.940599 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:28.049106 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c80d0ab-48a7-4cf1-a437-1af4bd016023 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:28.085666 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea2afabc-b9fc-4c5e-b99c-4560ed7de384 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:28.086348 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea2afabc-b9fc-4c5e-b99c-4560ed7de384 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:28.086629 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea2afabc-b9fc-4c5e-b99c-4560ed7de384 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:28.086907 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea2afabc-b9fc-4c5e-b99c-4560ed7de384 None admin] PATH_INFO: `/v3/projects/f6d8dbb619904d8889100044bad1bef6` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:28.088052 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea2afabc-b9fc-4c5e-b99c-4560ed7de384 None admin] RBAC: Authorizing `identity:get_project(project_id=f6d8dbb619904d8889100044bad1bef6)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:28.089503 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea2afabc-b9fc-4c5e-b99c-4560ed7de384 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:28.091525 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 202/409] 10.222.0.90 () {56 vars in 1186 bytes} [Mon Aug 29 21:20:27 2022] GET /identity/v3/projects/f6d8dbb619904d8889100044bad1bef6 => generated 302 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:28.097226 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f806c756-088e-4f69-abcc-ccea12835bfe None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:28.132491 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea2afabc-b9fc-4c5e-b99c-4560ed7de384 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:28.283325 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f806c756-088e-4f69-abcc-ccea12835bfe None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:28.315572 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:28.325428 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22164a71-9d0c-4c35-afd8-de4da2bf7dc4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:28.326719 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22164a71-9d0c-4c35-afd8-de4da2bf7dc4 None admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:28.327061 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22164a71-9d0c-4c35-afd8-de4da2bf7dc4 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:28.327294 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22164a71-9d0c-4c35-afd8-de4da2bf7dc4 None admin] PATH_INFO: `/v3/projects/f6d8dbb619904d8889100044bad1bef6/groups/792e5c9f64ca4f019cee6fd3fe995744/roles/9745586777bf473bab7d887bf25bf236` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:28.329710 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22164a71-9d0c-4c35-afd8-de4da2bf7dc4 None admin] RBAC: Authorizing `identity:create_grant(project_id=f6d8dbb619904d8889100044bad1bef6, group_id=792e5c9f64ca4f019cee6fd3fe995744, role_id=9745586777bf473bab7d887bf25bf236)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:28.331471 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22164a71-9d0c-4c35-afd8-de4da2bf7dc4 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:28.353065 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 208/410] 10.222.0.90 () {58 vars in 1442 bytes} [Mon Aug 29 21:20:28 2022] PUT /identity/v3/projects/f6d8dbb619904d8889100044bad1bef6/groups/792e5c9f64ca4f019cee6fd3fe995744/roles/9745586777bf473bab7d887bf25bf236 => generated 0 bytes in 256 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 21:20:28.372212 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea2afabc-b9fc-4c5e-b99c-4560ed7de384 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:28.439901 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53d1a206-bec0-4cba-854c-f37bf941de7f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:28.440462 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53d1a206-bec0-4cba-854c-f37bf941de7f None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:28.440739 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53d1a206-bec0-4cba-854c-f37bf941de7f None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:28.441047 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53d1a206-bec0-4cba-854c-f37bf941de7f None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:28.463049 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53d1a206-bec0-4cba-854c-f37bf941de7f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 21:20:28.467951 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-53d1a206-bec0-4cba-854c-f37bf941de7f None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 21:20:28.467951 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:28.467951 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:28.467951 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:28.467951 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:28.467951 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:28.467951 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:28.467951 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:28.467951 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:28.467951 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:28.467951 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:28.467951 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:28.467951 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 21:20:28.467951 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 21:20:28.467951 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 21:20:28.467951 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 21:20:28.468986 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 21:20:28.468986 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 21:20:28.468986 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 21:20:28.468986 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 21:20:28.468986 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:28.468986 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:28.468986 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:28.468986 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:28.468986 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:28.468986 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:28.468986 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:28.468986 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:28.468986 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:28.468986 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:28.468986 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:28.468986 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:28.469826 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:28.469826 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:28.469826 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:28.469826 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:28.469826 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 21:20:28.469826 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 21:20:28.469826 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 21:20:28.469826 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 21:20:28.469826 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 21:20:28.469826 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 21:20:28.469826 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 21:20:28.469826 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:20:28.472225 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 203/411] 10.222.0.90 () {56 vars in 1096 bytes} [Mon Aug 29 21:20:28 2022] GET /identity/v3/roles/admin => generated 83 bytes in 337 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 29 21:20:28.478157 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22164a71-9d0c-4c35-afd8-de4da2bf7dc4 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:28.545907 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dda43fd0-fdb3-4387-84c0-4312a091bd9d None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:28.546229 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dda43fd0-fdb3-4387-84c0-4312a091bd9d None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:28.546470 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dda43fd0-fdb3-4387-84c0-4312a091bd9d None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:28.548866 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 204/412] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:20:28 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 29 21:20:28.553431 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beed9f38-7bdd-4096-8d16-f06cd6da0e0f None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:28.553431 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beed9f38-7bdd-4096-8d16-f06cd6da0e0f None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:28.553652 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beed9f38-7bdd-4096-8d16-f06cd6da0e0f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:28.584890 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:28.590212 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-beed9f38-7bdd-4096-8d16-f06cd6da0e0f None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:20:28.591424 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-beed9f38-7bdd-4096-8d16-f06cd6da0e0f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:20:28.631595 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22164a71-9d0c-4c35-afd8-de4da2bf7dc4 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:28.682545 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 205/413] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:20:28 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 21:20:28.684179 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-783fb2e5-2b13-4f59-a924-d6d3ffe235a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:28.688797 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-783fb2e5-2b13-4f59-a924-d6d3ffe235a3 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:28.688797 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-783fb2e5-2b13-4f59-a924-d6d3ffe235a3 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:28.688797 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-783fb2e5-2b13-4f59-a924-d6d3ffe235a3 None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:28.688797 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-783fb2e5-2b13-4f59-a924-d6d3ffe235a3 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:28.688797 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-783fb2e5-2b13-4f59-a924-d6d3ffe235a3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:28.688797 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-783fb2e5-2b13-4f59-a924-d6d3ffe235a3 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:28.697181 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 209/414] 10.222.0.90 () {56 vars in 1099 bytes} [Mon Aug 29 21:20:28 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:28.702281 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beed9f38-7bdd-4096-8d16-f06cd6da0e0f None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:28.711148 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eae39f92-10b0-45d6-be75-acc1efb66ea2 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:28.711562 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eae39f92-10b0-45d6-be75-acc1efb66ea2 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:28.711943 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eae39f92-10b0-45d6-be75-acc1efb66ea2 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:28.713658 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 210/415] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:20:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 29 21:20:28.718325 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57420333-46af-4a0a-a833-121dc7c9ead7 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:28.718698 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57420333-46af-4a0a-a833-121dc7c9ead7 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:28.719071 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57420333-46af-4a0a-a833-121dc7c9ead7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:28.754364 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-57420333-46af-4a0a-a833-121dc7c9ead7 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:20:28.755740 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-57420333-46af-4a0a-a833-121dc7c9ead7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:20:28.794456 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:28.826206 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beed9f38-7bdd-4096-8d16-f06cd6da0e0f None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:28.867770 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0034335e-c9d9-44c1-813c-b5d766e9af2e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:28.868635 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0034335e-c9d9-44c1-813c-b5d766e9af2e None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:28.869064 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0034335e-c9d9-44c1-813c-b5d766e9af2e None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:28.869485 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0034335e-c9d9-44c1-813c-b5d766e9af2e None admin] PATH_INFO: `/v3/groups/1d230e348ec544faac7eeff5f0a65f0a` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:28.870921 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0034335e-c9d9-44c1-813c-b5d766e9af2e None admin] RBAC: Authorizing `identity:get_group(group_id=1d230e348ec544faac7eeff5f0a65f0a)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:28.872500 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0034335e-c9d9-44c1-813c-b5d766e9af2e None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:28.874743 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 206/416] 10.222.0.90 () {56 vars in 1180 bytes} [Mon Aug 29 21:20:28 2022] GET /identity/v3/groups/1d230e348ec544faac7eeff5f0a65f0a => generated 229 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:28.882158 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0034335e-c9d9-44c1-813c-b5d766e9af2e None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:28.999540 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 211/417] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:20:28 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 279 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 21:20:29.007166 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57420333-46af-4a0a-a833-121dc7c9ead7 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:29.046256 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0034335e-c9d9-44c1-813c-b5d766e9af2e None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:29.086165 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:29.123321 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57420333-46af-4a0a-a833-121dc7c9ead7 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:29.161310 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f30fbce0-98aa-4547-ba6c-a18a46255e6f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:29.162221 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f30fbce0-98aa-4547-ba6c-a18a46255e6f None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:29.163094 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f30fbce0-98aa-4547-ba6c-a18a46255e6f None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:29.163552 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f30fbce0-98aa-4547-ba6c-a18a46255e6f None admin] PATH_INFO: `/v3/roles/member` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:29.169437 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50536b56-92b8-46d2-b8e8-fdf4c42ae8b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:29.172350 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50536b56-92b8-46d2-b8e8-fdf4c42ae8b1 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:29.172783 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50536b56-92b8-46d2-b8e8-fdf4c42ae8b1 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:29.173221 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50536b56-92b8-46d2-b8e8-fdf4c42ae8b1 None admin] PATH_INFO: `/v3/projects/fa1ebaa0d49241599f614eb2efe4d8c8` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:29.174569 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50536b56-92b8-46d2-b8e8-fdf4c42ae8b1 None admin] RBAC: Authorizing `identity:get_project(project_id=fa1ebaa0d49241599f614eb2efe4d8c8)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:29.177872 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50536b56-92b8-46d2-b8e8-fdf4c42ae8b1 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:29.187121 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 207/418] 10.222.0.90 () {56 vars in 1186 bytes} [Mon Aug 29 21:20:28 2022] GET /identity/v3/projects/fa1ebaa0d49241599f614eb2efe4d8c8 => generated 344 bytes in 305 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:29.187823 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50536b56-92b8-46d2-b8e8-fdf4c42ae8b1 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:29.190137 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f30fbce0-98aa-4547-ba6c-a18a46255e6f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 21:20:29.195945 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-f30fbce0-98aa-4547-ba6c-a18a46255e6f None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 29 21:20:29.195945 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:29.195945 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:29.195945 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:29.195945 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:29.195945 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:29.195945 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:29.195945 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:29.195945 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:29.195945 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:29.195945 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:29.195945 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:29.195945 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 21:20:29.195945 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 21:20:29.195945 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 21:20:29.195945 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 21:20:29.197130 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 21:20:29.197130 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 21:20:29.197130 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 21:20:29.197130 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 21:20:29.197130 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:29.197130 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:29.197130 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:29.197130 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:29.197130 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:29.197130 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:29.197130 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:29.197130 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:29.197130 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:29.197130 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:29.197130 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:29.197130 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:29.198175 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:29.198175 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:29.198175 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:29.198175 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:29.198175 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 21:20:29.198175 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 21:20:29.198175 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 21:20:29.198175 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 21:20:29.198175 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 21:20:29.198175 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 21:20:29.198175 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 29 21:20:29.198175 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:20:29.200745 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 212/419] 10.222.0.90 () {56 vars in 1099 bytes} [Mon Aug 29 21:20:29 2022] GET /identity/v3/roles/member => generated 84 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 29 21:20:29.206257 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f30fbce0-98aa-4547-ba6c-a18a46255e6f None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:29.223776 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:29.322500 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f30fbce0-98aa-4547-ba6c-a18a46255e6f None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:29.335594 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:29.378489 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9eb22be-6347-4472-959b-3eec1f0fdae0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:29.379327 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9eb22be-6347-4472-959b-3eec1f0fdae0 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:29.379686 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9eb22be-6347-4472-959b-3eec1f0fdae0 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:29.379958 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9eb22be-6347-4472-959b-3eec1f0fdae0 None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:29.380658 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9eb22be-6347-4472-959b-3eec1f0fdae0 None admin] RBAC: Adding query filter params (name=member) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:29.380961 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9eb22be-6347-4472-959b-3eec1f0fdae0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:29.382368 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9eb22be-6347-4472-959b-3eec1f0fdae0 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:29.391676 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 213/420] 10.222.0.90 () {56 vars in 1101 bytes} [Mon Aug 29 21:20:29 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:20:29.392513 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9eb22be-6347-4472-959b-3eec1f0fdae0 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:29.406245 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50536b56-92b8-46d2-b8e8-fdf4c42ae8b1 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:29.456301 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ac347c8-5e06-4797-8e05-7f6e84a43589 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:29.458253 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ac347c8-5e06-4797-8e05-7f6e84a43589 None admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:29.458939 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ac347c8-5e06-4797-8e05-7f6e84a43589 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:29.459353 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ac347c8-5e06-4797-8e05-7f6e84a43589 None admin] PATH_INFO: `/v3/projects/fa1ebaa0d49241599f614eb2efe4d8c8/groups/1d230e348ec544faac7eeff5f0a65f0a/roles/a6e562cdc96c4be691d638990f8403fd` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:29.462426 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ac347c8-5e06-4797-8e05-7f6e84a43589 None admin] RBAC: Authorizing `identity:create_grant(project_id=fa1ebaa0d49241599f614eb2efe4d8c8, group_id=1d230e348ec544faac7eeff5f0a65f0a, role_id=a6e562cdc96c4be691d638990f8403fd)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:29.464656 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ac347c8-5e06-4797-8e05-7f6e84a43589 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:29.481743 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 208/421] 10.222.0.90 () {58 vars in 1442 bytes} [Mon Aug 29 21:20:29 2022] PUT /identity/v3/projects/fa1ebaa0d49241599f614eb2efe4d8c8/groups/1d230e348ec544faac7eeff5f0a65f0a/roles/a6e562cdc96c4be691d638990f8403fd => generated 0 bytes in 294 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:20:29.483365 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ac347c8-5e06-4797-8e05-7f6e84a43589 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:29.510342 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9eb22be-6347-4472-959b-3eec1f0fdae0 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:29.554478 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae67d23b-89c3-4e10-8cb9-a5423513e84a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:29.555383 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae67d23b-89c3-4e10-8cb9-a5423513e84a None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:29.555760 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae67d23b-89c3-4e10-8cb9-a5423513e84a None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:29.556129 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae67d23b-89c3-4e10-8cb9-a5423513e84a None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:29.572387 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae67d23b-89c3-4e10-8cb9-a5423513e84a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 21:20:29.577728 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:29.582816 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-ae67d23b-89c3-4e10-8cb9-a5423513e84a None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 29 21:20:29.582816 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:29.582816 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:29.582816 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:29.582816 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:29.582816 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:29.582816 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:29.582816 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:29.582816 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:29.582816 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:29.582816 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:29.582816 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:29.582816 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 21:20:29.582816 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 21:20:29.582816 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 21:20:29.582816 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 21:20:29.583852 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 21:20:29.583852 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 21:20:29.583852 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 21:20:29.583852 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 21:20:29.583852 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:29.583852 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:29.583852 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:29.583852 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:29.583852 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:29.583852 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:29.583852 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:29.583852 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:29.583852 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:29.583852 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:29.583852 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:29.583852 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:29.584673 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:29.584673 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:29.584673 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:29.584673 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:29.584673 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 21:20:29.584673 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 21:20:29.584673 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 21:20:29.584673 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 21:20:29.584673 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 21:20:29.584673 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 21:20:29.584673 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 29 21:20:29.584673 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:20:29.584673 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 214/422] 10.222.0.90 () {56 vars in 1114 bytes} [Mon Aug 29 21:20:29 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:20:29.597688 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae67d23b-89c3-4e10-8cb9-a5423513e84a None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:29.655766 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ac347c8-5e06-4797-8e05-7f6e84a43589 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:29.658221 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:29.699347 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d35c94f-8c2d-4c6b-bb75-9e7e0f8f1c13 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:29.699969 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d35c94f-8c2d-4c6b-bb75-9e7e0f8f1c13 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:29.700244 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d35c94f-8c2d-4c6b-bb75-9e7e0f8f1c13 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:29.700512 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d35c94f-8c2d-4c6b-bb75-9e7e0f8f1c13 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:29.767362 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae67d23b-89c3-4e10-8cb9-a5423513e84a None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:29.798806 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d35c94f-8c2d-4c6b-bb75-9e7e0f8f1c13 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:29.800267 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d35c94f-8c2d-4c6b-bb75-9e7e0f8f1c13 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:29.829163 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a22cea42-b84a-4d1d-abce-c7c97cdbbfe4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:29.829699 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a22cea42-b84a-4d1d-abce-c7c97cdbbfe4 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:29.829999 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a22cea42-b84a-4d1d-abce-c7c97cdbbfe4 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:29.830268 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a22cea42-b84a-4d1d-abce-c7c97cdbbfe4 None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:29.830935 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a22cea42-b84a-4d1d-abce-c7c97cdbbfe4 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:29.831234 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a22cea42-b84a-4d1d-abce-c7c97cdbbfe4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:29.833600 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a22cea42-b84a-4d1d-abce-c7c97cdbbfe4 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:29.850735 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 215/423] 10.222.0.90 () {56 vars in 1111 bytes} [Mon Aug 29 21:20:29 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 254 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:20:29.874105 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a22cea42-b84a-4d1d-abce-c7c97cdbbfe4 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:29.934467 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:29.977726 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:29.998062 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a22cea42-b84a-4d1d-abce-c7c97cdbbfe4 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:30.037327 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 209/424] 10.222.0.90 () {58 vars in 1282 bytes} [Mon Aug 29 21:20:29 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 555 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 29 21:20:30.043332 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d35c94f-8c2d-4c6b-bb75-9e7e0f8f1c13 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:30.050588 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10c84ddf-bd7c-4706-abb6-3f2cdb06ebe2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:30.052075 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10c84ddf-bd7c-4706-abb6-3f2cdb06ebe2 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:30.052522 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10c84ddf-bd7c-4706-abb6-3f2cdb06ebe2 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:30.052938 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10c84ddf-bd7c-4706-abb6-3f2cdb06ebe2 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:30.186941 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10c84ddf-bd7c-4706-abb6-3f2cdb06ebe2 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:30.189611 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10c84ddf-bd7c-4706-abb6-3f2cdb06ebe2 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:30.198078 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d35c94f-8c2d-4c6b-bb75-9e7e0f8f1c13 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:30.246428 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9a91c61-d559-4b71-a5e3-4404ad0df17b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:30.247878 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9a91c61-d559-4b71-a5e3-4404ad0df17b None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:30.248301 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9a91c61-d559-4b71-a5e3-4404ad0df17b None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:30.248992 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9a91c61-d559-4b71-a5e3-4404ad0df17b None admin] PATH_INFO: `/v3/projects/381cc34791e147329798cd454f99f622` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:30.250646 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9a91c61-d559-4b71-a5e3-4404ad0df17b None admin] RBAC: Authorizing `identity:get_project(project_id=381cc34791e147329798cd454f99f622)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:30.252945 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9a91c61-d559-4b71-a5e3-4404ad0df17b None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:30.256761 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 210/425] 10.222.0.90 () {56 vars in 1186 bytes} [Mon Aug 29 21:20:30 2022] GET /identity/v3/projects/381cc34791e147329798cd454f99f622 => generated 298 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:30.258855 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee3f01e9-7c61-4808-ba9d-90f285571ca2 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:30.259302 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee3f01e9-7c61-4808-ba9d-90f285571ca2 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:30.259714 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee3f01e9-7c61-4808-ba9d-90f285571ca2 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:30.265011 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 211/426] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:20:30 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:20:30.265011 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee3f01e9-7c61-4808-ba9d-90f285571ca2 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:30.292622 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:30.313514 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:30.319814 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 216/427] 10.222.0.90 () {58 vars in 1282 bytes} [Mon Aug 29 21:20:29 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 441 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 29 21:20:30.319814 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-839a2c5c-ddb6-43f1-b659-8b0ee7de2ba5 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:30.319814 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-839a2c5c-ddb6-43f1-b659-8b0ee7de2ba5 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:30.319814 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-839a2c5c-ddb6-43f1-b659-8b0ee7de2ba5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:30.337767 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-839a2c5c-ddb6-43f1-b659-8b0ee7de2ba5 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:20:30.339342 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-839a2c5c-ddb6-43f1-b659-8b0ee7de2ba5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:20:30.405308 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee3f01e9-7c61-4808-ba9d-90f285571ca2 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:30.417810 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 217/428] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:20:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 21:20:30.417810 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-839a2c5c-ddb6-43f1-b659-8b0ee7de2ba5 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:30.459605 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5c02c91-5cf7-409d-a5d2-de219bce5e06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:30.463901 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5c02c91-5cf7-409d-a5d2-de219bce5e06 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:30.464384 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5c02c91-5cf7-409d-a5d2-de219bce5e06 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:30.464851 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5c02c91-5cf7-409d-a5d2-de219bce5e06 None admin] PATH_INFO: `/v3/groups/792e5c9f64ca4f019cee6fd3fe995744` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:30.466260 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5c02c91-5cf7-409d-a5d2-de219bce5e06 None admin] RBAC: Authorizing `identity:get_group(group_id=792e5c9f64ca4f019cee6fd3fe995744)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:30.467894 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5c02c91-5cf7-409d-a5d2-de219bce5e06 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:30.473003 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 212/429] 10.222.0.90 () {56 vars in 1180 bytes} [Mon Aug 29 21:20:30 2022] GET /identity/v3/groups/792e5c9f64ca4f019cee6fd3fe995744 => generated 226 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:30.479441 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5c02c91-5cf7-409d-a5d2-de219bce5e06 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:30.594957 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-839a2c5c-ddb6-43f1-b659-8b0ee7de2ba5 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:30.613695 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:30.642170 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12d7e326-a741-47ac-9687-8c50624b98f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:30.642920 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5c02c91-5cf7-409d-a5d2-de219bce5e06 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:30.643043 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12d7e326-a741-47ac-9687-8c50624b98f2 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:30.643427 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12d7e326-a741-47ac-9687-8c50624b98f2 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:30.643798 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12d7e326-a741-47ac-9687-8c50624b98f2 None admin] PATH_INFO: `/v3/projects/381cc34791e147329798cd454f99f622` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:30.646153 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12d7e326-a741-47ac-9687-8c50624b98f2 None admin] RBAC: Authorizing `identity:get_project(project_id=381cc34791e147329798cd454f99f622)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:30.648620 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12d7e326-a741-47ac-9687-8c50624b98f2 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:30.650692 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 218/430] 10.222.0.90 () {56 vars in 1186 bytes} [Mon Aug 29 21:20:30 2022] GET /identity/v3/projects/381cc34791e147329798cd454f99f622 => generated 298 bytes in 241 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:30.651928 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0bf15b6-73f2-44f0-93fb-eea7b82531af None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:30.654108 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0bf15b6-73f2-44f0-93fb-eea7b82531af None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:30.654588 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0bf15b6-73f2-44f0-93fb-eea7b82531af None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:30.655979 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 219/431] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:20:30 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:20:30.659967 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:30.661021 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0bf15b6-73f2-44f0-93fb-eea7b82531af None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:30.694643 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f0d2cc4-62ac-48fe-ab62-c8153914390d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:30.695355 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f0d2cc4-62ac-48fe-ab62-c8153914390d None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:30.695994 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f0d2cc4-62ac-48fe-ab62-c8153914390d None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:30.696341 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f0d2cc4-62ac-48fe-ab62-c8153914390d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:30.697399 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f0d2cc4-62ac-48fe-ab62-c8153914390d None admin] RBAC: Adding query filter params (group.id=792e5c9f64ca4f019cee6fd3fe995744, role.id=02e242d5520740d4b77a5f4008c7eb57, scope.project.id=381cc34791e147329798cd454f99f622) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:30.697815 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f0d2cc4-62ac-48fe-ab62-c8153914390d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:30.699300 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f0d2cc4-62ac-48fe-ab62-c8153914390d None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:30.717964 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 213/432] 10.222.0.90 () {56 vars in 1376 bytes} [Mon Aug 29 21:20:30 2022] GET /identity/v3/role_assignments?group.id=792e5c9f64ca4f019cee6fd3fe995744&scope.project.id=381cc34791e147329798cd454f99f622&role.id=02e242d5520740d4b77a5f4008c7eb57 => generated 611 bytes in 231 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:30.717964 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5231485e-6298-40b6-9f62-6ccf5035c8c1 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:30.717964 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5231485e-6298-40b6-9f62-6ccf5035c8c1 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:30.717964 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5231485e-6298-40b6-9f62-6ccf5035c8c1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:30.732078 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-5231485e-6298-40b6-9f62-6ccf5035c8c1 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:20:30.733386 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-5231485e-6298-40b6-9f62-6ccf5035c8c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:20:30.796537 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0bf15b6-73f2-44f0-93fb-eea7b82531af None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:30.813811 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 214/433] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:20:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 21:20:30.814690 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5231485e-6298-40b6-9f62-6ccf5035c8c1 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:30.850619 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9425d5dc-5526-4278-9930-d66b3449b17e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:30.851565 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9425d5dc-5526-4278-9930-d66b3449b17e None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:30.851848 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9425d5dc-5526-4278-9930-d66b3449b17e None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:30.852119 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9425d5dc-5526-4278-9930-d66b3449b17e None admin] PATH_INFO: `/v3/groups/792e5c9f64ca4f019cee6fd3fe995744` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:30.853332 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9425d5dc-5526-4278-9930-d66b3449b17e None admin] RBAC: Authorizing `identity:get_group(group_id=792e5c9f64ca4f019cee6fd3fe995744)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:30.854801 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9425d5dc-5526-4278-9930-d66b3449b17e None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:30.856683 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 220/434] 10.222.0.90 () {56 vars in 1180 bytes} [Mon Aug 29 21:20:30 2022] GET /identity/v3/groups/792e5c9f64ca4f019cee6fd3fe995744 => generated 226 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:30.866834 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9425d5dc-5526-4278-9930-d66b3449b17e None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:30.938748 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:30.938951 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:30.944042 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5231485e-6298-40b6-9f62-6ccf5035c8c1 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:30.981175 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4091660d-15ea-423f-8bc4-0ba92891816b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:30.982086 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4091660d-15ea-423f-8bc4-0ba92891816b None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:30.982557 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4091660d-15ea-423f-8bc4-0ba92891816b None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:30.983014 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4091660d-15ea-423f-8bc4-0ba92891816b None admin] PATH_INFO: `/v3/roles/member` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:30.994685 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9425d5dc-5526-4278-9930-d66b3449b17e None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:30.995211 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4091660d-15ea-423f-8bc4-0ba92891816b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 21:20:31.000226 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-4091660d-15ea-423f-8bc4-0ba92891816b None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 29 21:20:31.000226 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:31.000226 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:31.000226 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:31.000226 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:31.000226 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:31.000226 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:31.000226 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:31.000226 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:31.000226 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:31.000226 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:31.000226 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:31.000226 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 21:20:31.000226 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 21:20:31.000226 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 21:20:31.000226 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 21:20:31.001378 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 21:20:31.001378 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 21:20:31.001378 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 21:20:31.001378 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 21:20:31.001378 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:31.001378 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:31.001378 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:31.001378 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:31.001378 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:31.001378 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:31.001378 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:31.001378 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:31.001378 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:31.001378 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:31.001378 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:31.001378 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:31.002346 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:31.002346 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:31.002346 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:31.002346 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:31.002346 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 21:20:31.002346 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 21:20:31.002346 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 21:20:31.002346 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 21:20:31.002346 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 21:20:31.002346 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 21:20:31.002346 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 29 21:20:31.002346 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:20:31.005034 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 215/435] 10.222.0.90 () {56 vars in 1099 bytes} [Mon Aug 29 21:20:30 2022] GET /identity/v3/roles/member => generated 84 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:20:31.015074 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4091660d-15ea-423f-8bc4-0ba92891816b None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:31.046819 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19d61fc3-77cb-410f-abb0-a4c17bd6aa1c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:31.047532 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d61fc3-77cb-410f-abb0-a4c17bd6aa1c None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:31.047816 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d61fc3-77cb-410f-abb0-a4c17bd6aa1c None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:31.048086 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d61fc3-77cb-410f-abb0-a4c17bd6aa1c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:31.048918 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19d61fc3-77cb-410f-abb0-a4c17bd6aa1c None admin] RBAC: Adding query filter params (group.id=792e5c9f64ca4f019cee6fd3fe995744, role.id=9745586777bf473bab7d887bf25bf236, scope.project.id=381cc34791e147329798cd454f99f622) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:31.049228 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19d61fc3-77cb-410f-abb0-a4c17bd6aa1c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:31.050806 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19d61fc3-77cb-410f-abb0-a4c17bd6aa1c None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:31.058308 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 221/436] 10.222.0.90 () {56 vars in 1376 bytes} [Mon Aug 29 21:20:30 2022] GET /identity/v3/role_assignments?group.id=792e5c9f64ca4f019cee6fd3fe995744&scope.project.id=381cc34791e147329798cd454f99f622&role.id=9745586777bf473bab7d887bf25bf236 => generated 611 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:31.120290 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19d61fc3-77cb-410f-abb0-a4c17bd6aa1c None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:31.136608 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4091660d-15ea-423f-8bc4-0ba92891816b None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:31.180726 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36fd5ee0-437c-45fc-9784-aa932238ab6f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:31.181604 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36fd5ee0-437c-45fc-9784-aa932238ab6f None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:31.182057 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36fd5ee0-437c-45fc-9784-aa932238ab6f None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:31.182500 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36fd5ee0-437c-45fc-9784-aa932238ab6f None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:31.183372 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36fd5ee0-437c-45fc-9784-aa932238ab6f None admin] RBAC: Adding query filter params (name=member) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:31.183818 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36fd5ee0-437c-45fc-9784-aa932238ab6f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:31.185365 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36fd5ee0-437c-45fc-9784-aa932238ab6f None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:31.198289 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 216/437] 10.222.0.90 () {56 vars in 1101 bytes} [Mon Aug 29 21:20:31 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:31.211020 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36fd5ee0-437c-45fc-9784-aa932238ab6f None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:31.253206 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19d61fc3-77cb-410f-abb0-a4c17bd6aa1c None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:31.254773 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:31.272400 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:31.298025 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a7e0b3d-1309-4c23-844a-e7119d545a0e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:31.298652 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a7e0b3d-1309-4c23-844a-e7119d545a0e None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:31.298930 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a7e0b3d-1309-4c23-844a-e7119d545a0e None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:31.299203 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a7e0b3d-1309-4c23-844a-e7119d545a0e None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:31.313201 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a7e0b3d-1309-4c23-844a-e7119d545a0e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 21:20:31.322039 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-4a7e0b3d-1309-4c23-844a-e7119d545a0e None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 29 21:20:31.322039 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:31.322039 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:31.322039 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:31.322039 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:31.322039 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:31.322039 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:31.322039 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:31.322039 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:31.322039 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:31.322039 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:31.322039 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:31.322039 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 21:20:31.322039 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 21:20:31.322039 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 21:20:31.322039 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 21:20:31.322999 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 21:20:31.322999 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 21:20:31.322999 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 21:20:31.322999 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 21:20:31.322999 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:31.322999 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:31.322999 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:31.322999 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:31.322999 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:31.322999 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:31.322999 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:31.322999 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:31.322999 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:31.322999 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:31.322999 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:31.322999 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:31.323841 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:31.323841 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:31.323841 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:31.323841 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:31.323841 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 21:20:31.323841 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 21:20:31.323841 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 21:20:31.323841 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 21:20:31.323841 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 21:20:31.323841 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 21:20:31.323841 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 29 21:20:31.323841 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:20:31.323841 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 222/438] 10.222.0.90 () {56 vars in 1114 bytes} [Mon Aug 29 21:20:31 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 29 21:20:31.327779 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a7e0b3d-1309-4c23-844a-e7119d545a0e None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:31.367610 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36fd5ee0-437c-45fc-9784-aa932238ab6f None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:31.420376 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de6779d2-1479-4812-af3e-b05e9ba72fbb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:31.420999 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de6779d2-1479-4812-af3e-b05e9ba72fbb None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:31.421593 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de6779d2-1479-4812-af3e-b05e9ba72fbb None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:31.422843 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de6779d2-1479-4812-af3e-b05e9ba72fbb None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:31.451409 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a7e0b3d-1309-4c23-844a-e7119d545a0e None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:31.495820 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b55718c1-2385-431e-a27e-8fc2a9351995 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:31.496447 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b55718c1-2385-431e-a27e-8fc2a9351995 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:31.496727 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b55718c1-2385-431e-a27e-8fc2a9351995 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:31.496999 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b55718c1-2385-431e-a27e-8fc2a9351995 None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:31.497678 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b55718c1-2385-431e-a27e-8fc2a9351995 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:31.498014 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b55718c1-2385-431e-a27e-8fc2a9351995 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:31.499386 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b55718c1-2385-431e-a27e-8fc2a9351995 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:31.509043 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 223/439] 10.222.0.90 () {56 vars in 1111 bytes} [Mon Aug 29 21:20:31 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:31.513340 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de6779d2-1479-4812-af3e-b05e9ba72fbb None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:31.514999 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b55718c1-2385-431e-a27e-8fc2a9351995 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:31.515490 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de6779d2-1479-4812-af3e-b05e9ba72fbb None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:31.555407 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:31.571169 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:31.653890 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 217/440] 10.222.0.90 () {58 vars in 1282 bytes} [Mon Aug 29 21:20:31 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 440 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 29 21:20:31.655867 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6baeedc-8d65-404b-965a-2f029f70fe15 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:31.656201 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6baeedc-8d65-404b-965a-2f029f70fe15 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:31.656480 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6baeedc-8d65-404b-965a-2f029f70fe15 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:31.657830 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 218/441] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:20:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:20:31.658543 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6baeedc-8d65-404b-965a-2f029f70fe15 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:31.664180 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b55718c1-2385-431e-a27e-8fc2a9351995 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:31.706263 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c30b4ff1-6b31-4a83-89cb-8deac6147738 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:31.707409 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c30b4ff1-6b31-4a83-89cb-8deac6147738 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:31.707777 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c30b4ff1-6b31-4a83-89cb-8deac6147738 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:31.708076 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c30b4ff1-6b31-4a83-89cb-8deac6147738 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:31.776778 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6baeedc-8d65-404b-965a-2f029f70fe15 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:31.781517 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c30b4ff1-6b31-4a83-89cb-8deac6147738 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:31.783028 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c30b4ff1-6b31-4a83-89cb-8deac6147738 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:31.800356 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:31.812504 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7607988c-3d09-42d4-8b37-c884ede88a47 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:31.813137 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7607988c-3d09-42d4-8b37-c884ede88a47 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:31.813418 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7607988c-3d09-42d4-8b37-c884ede88a47 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:31.813701 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7607988c-3d09-42d4-8b37-c884ede88a47 None admin] PATH_INFO: `/v3/projects/f6d8dbb619904d8889100044bad1bef6` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:31.814849 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7607988c-3d09-42d4-8b37-c884ede88a47 None admin] RBAC: Authorizing `identity:get_project(project_id=f6d8dbb619904d8889100044bad1bef6)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:31.816903 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7607988c-3d09-42d4-8b37-c884ede88a47 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:31.817443 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:31.818827 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 219/442] 10.222.0.90 () {56 vars in 1186 bytes} [Mon Aug 29 21:20:31 2022] GET /identity/v3/projects/f6d8dbb619904d8889100044bad1bef6 => generated 302 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:31.819952 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9769a54c-4087-4ea2-b403-af09119ebe71 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:31.820219 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9769a54c-4087-4ea2-b403-af09119ebe71 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:31.820498 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9769a54c-4087-4ea2-b403-af09119ebe71 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:31.838793 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-9769a54c-4087-4ea2-b403-af09119ebe71 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:20:31.839992 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-9769a54c-4087-4ea2-b403-af09119ebe71 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:20:31.885317 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 224/443] 10.222.0.90 () {58 vars in 1282 bytes} [Mon Aug 29 21:20:31 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 371 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Aug 29 21:20:31.886281 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c30b4ff1-6b31-4a83-89cb-8deac6147738 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:31.912963 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 220/444] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:20:31 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 21:20:31.913840 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9769a54c-4087-4ea2-b403-af09119ebe71 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:32.001844 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c30b4ff1-6b31-4a83-89cb-8deac6147738 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:32.020332 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9769a54c-4087-4ea2-b403-af09119ebe71 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:32.039991 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65e9c6f2-082a-480e-8c82-84873d876f0b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:32.040633 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65e9c6f2-082a-480e-8c82-84873d876f0b None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:32.041199 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65e9c6f2-082a-480e-8c82-84873d876f0b None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:32.041199 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65e9c6f2-082a-480e-8c82-84873d876f0b None admin] PATH_INFO: `/v3/groups/792e5c9f64ca4f019cee6fd3fe995744` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:32.042319 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65e9c6f2-082a-480e-8c82-84873d876f0b None admin] RBAC: Authorizing `identity:get_group(group_id=792e5c9f64ca4f019cee6fd3fe995744)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:32.042652 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:32.043761 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65e9c6f2-082a-480e-8c82-84873d876f0b None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:32.045485 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 225/445] 10.222.0.90 () {56 vars in 1180 bytes} [Mon Aug 29 21:20:31 2022] GET /identity/v3/groups/792e5c9f64ca4f019cee6fd3fe995744 => generated 226 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:32.051388 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65e9c6f2-082a-480e-8c82-84873d876f0b None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:32.055793 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:32.057258 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6988395e-3e23-4ef8-935a-dda49ec31013 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:32.058002 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6988395e-3e23-4ef8-935a-dda49ec31013 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:32.058289 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6988395e-3e23-4ef8-935a-dda49ec31013 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:32.058616 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6988395e-3e23-4ef8-935a-dda49ec31013 None admin] PATH_INFO: `/v3/projects/f6d8dbb619904d8889100044bad1bef6` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:32.059665 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6988395e-3e23-4ef8-935a-dda49ec31013 None admin] RBAC: Authorizing `identity:get_project(project_id=f6d8dbb619904d8889100044bad1bef6)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:32.061094 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6988395e-3e23-4ef8-935a-dda49ec31013 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:32.062922 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 221/446] 10.222.0.90 () {56 vars in 1186 bytes} [Mon Aug 29 21:20:31 2022] GET /identity/v3/projects/f6d8dbb619904d8889100044bad1bef6 => generated 302 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:32.069094 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6988395e-3e23-4ef8-935a-dda49ec31013 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:32.164112 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65e9c6f2-082a-480e-8c82-84873d876f0b None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:32.171114 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6988395e-3e23-4ef8-935a-dda49ec31013 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:32.200290 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6707d03a-1e0b-4561-9218-263d1f4978a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:32.200901 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6707d03a-1e0b-4561-9218-263d1f4978a4 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:32.201180 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6707d03a-1e0b-4561-9218-263d1f4978a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:32.201462 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6707d03a-1e0b-4561-9218-263d1f4978a4 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:32.202218 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6707d03a-1e0b-4561-9218-263d1f4978a4 None admin] RBAC: Adding query filter params (group.id=792e5c9f64ca4f019cee6fd3fe995744, role.id=02e242d5520740d4b77a5f4008c7eb57, scope.project.id=f6d8dbb619904d8889100044bad1bef6) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:32.202529 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6707d03a-1e0b-4561-9218-263d1f4978a4 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:32.203953 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6707d03a-1e0b-4561-9218-263d1f4978a4 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:32.205971 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-388317ec-3d87-48dc-8aa4-a203fb329b20 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:32.206926 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-388317ec-3d87-48dc-8aa4-a203fb329b20 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:32.206926 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-388317ec-3d87-48dc-8aa4-a203fb329b20 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:32.207182 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-388317ec-3d87-48dc-8aa4-a203fb329b20 None admin] PATH_INFO: `/v3/groups/792e5c9f64ca4f019cee6fd3fe995744` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:32.208286 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-388317ec-3d87-48dc-8aa4-a203fb329b20 None admin] RBAC: Authorizing `identity:get_group(group_id=792e5c9f64ca4f019cee6fd3fe995744)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:32.209693 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 226/447] 10.222.0.90 () {56 vars in 1376 bytes} [Mon Aug 29 21:20:32 2022] GET /identity/v3/role_assignments?group.id=792e5c9f64ca4f019cee6fd3fe995744&scope.project.id=f6d8dbb619904d8889100044bad1bef6&role.id=02e242d5520740d4b77a5f4008c7eb57 => generated 611 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:32.211865 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-388317ec-3d87-48dc-8aa4-a203fb329b20 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:32.211962 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6707d03a-1e0b-4561-9218-263d1f4978a4 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:32.212047 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 222/448] 10.222.0.90 () {56 vars in 1180 bytes} [Mon Aug 29 21:20:32 2022] GET /identity/v3/groups/792e5c9f64ca4f019cee6fd3fe995744 => generated 226 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:20:32.217369 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-388317ec-3d87-48dc-8aa4-a203fb329b20 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:32.285475 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:32.301403 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:32.338131 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-388317ec-3d87-48dc-8aa4-a203fb329b20 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:32.339535 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6707d03a-1e0b-4561-9218-263d1f4978a4 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:32.382824 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d24f498b-77c3-4e7d-a7be-03714f5ceec9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:32.383470 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d24f498b-77c3-4e7d-a7be-03714f5ceec9 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:32.383756 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d24f498b-77c3-4e7d-a7be-03714f5ceec9 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:32.384026 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d24f498b-77c3-4e7d-a7be-03714f5ceec9 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:32.384254 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0c007d3-b25e-4fd4-9201-733e5773b719 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:32.384876 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0c007d3-b25e-4fd4-9201-733e5773b719 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:32.385675 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0c007d3-b25e-4fd4-9201-733e5773b719 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:32.385675 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0c007d3-b25e-4fd4-9201-733e5773b719 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:32.386175 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0c007d3-b25e-4fd4-9201-733e5773b719 None admin] RBAC: Adding query filter params (group.id=792e5c9f64ca4f019cee6fd3fe995744, role.id=9745586777bf473bab7d887bf25bf236, scope.project.id=f6d8dbb619904d8889100044bad1bef6) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:32.386482 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0c007d3-b25e-4fd4-9201-733e5773b719 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:32.387924 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0c007d3-b25e-4fd4-9201-733e5773b719 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:32.395022 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 223/449] 10.222.0.90 () {56 vars in 1376 bytes} [Mon Aug 29 21:20:32 2022] GET /identity/v3/role_assignments?group.id=792e5c9f64ca4f019cee6fd3fe995744&scope.project.id=f6d8dbb619904d8889100044bad1bef6&role.id=9745586777bf473bab7d887bf25bf236 => generated 611 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:20:32.401513 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d24f498b-77c3-4e7d-a7be-03714f5ceec9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 21:20:32.407344 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-d24f498b-77c3-4e7d-a7be-03714f5ceec9 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 21:20:32.407344 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:32.407344 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:32.407344 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:32.407344 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:32.407344 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:32.407344 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:32.407344 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:32.407344 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:32.407344 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:32.407344 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:32.407344 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:32.407344 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 21:20:32.407344 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 21:20:32.407344 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 21:20:32.407344 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 21:20:32.408331 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 21:20:32.408331 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 21:20:32.408331 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 21:20:32.408331 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 21:20:32.408331 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:32.408331 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:32.408331 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:32.408331 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:32.408331 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:32.408331 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:32.408331 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:32.408331 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:32.408331 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:32.408331 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:32.408331 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:32.408331 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:32.409191 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:32.409191 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:32.409191 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:32.409191 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:32.409191 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 21:20:32.409191 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 21:20:32.409191 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 21:20:32.409191 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 21:20:32.409191 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 21:20:32.409191 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 21:20:32.409191 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 21:20:32.409191 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:20:32.409191 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 227/450] 10.222.0.90 () {56 vars in 1096 bytes} [Mon Aug 29 21:20:32 2022] GET /identity/v3/roles/admin => generated 83 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:20:32.416421 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0c007d3-b25e-4fd4-9201-733e5773b719 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:32.540813 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0c007d3-b25e-4fd4-9201-733e5773b719 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:32.561749 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:32.577927 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-090703a3-5c62-4d75-9bfe-12ddd8a02e7a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:32.578553 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-090703a3-5c62-4d75-9bfe-12ddd8a02e7a None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:32.578834 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-090703a3-5c62-4d75-9bfe-12ddd8a02e7a None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:32.579111 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-090703a3-5c62-4d75-9bfe-12ddd8a02e7a None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:32.579786 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-090703a3-5c62-4d75-9bfe-12ddd8a02e7a None admin] RBAC: Adding query filter params (name=admin) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:32.580091 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-090703a3-5c62-4d75-9bfe-12ddd8a02e7a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:32.581468 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-090703a3-5c62-4d75-9bfe-12ddd8a02e7a None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:32.591328 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 224/451] 10.222.0.90 () {56 vars in 1099 bytes} [Mon Aug 29 21:20:32 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:32.596863 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d24f498b-77c3-4e7d-a7be-03714f5ceec9 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:32.700853 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d24f498b-77c3-4e7d-a7be-03714f5ceec9 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:32.727389 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:32.735596 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee6a7e73-889f-4808-b88a-1d989a609faf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:32.736206 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee6a7e73-889f-4808-b88a-1d989a609faf None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:32.736480 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee6a7e73-889f-4808-b88a-1d989a609faf None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:32.736750 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee6a7e73-889f-4808-b88a-1d989a609faf None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:32.804658 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee6a7e73-889f-4808-b88a-1d989a609faf None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:32.806200 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee6a7e73-889f-4808-b88a-1d989a609faf None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:32.903073 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 228/452] 10.222.0.90 () {58 vars in 1282 bytes} [Mon Aug 29 21:20:32 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Aug 29 21:20:32.909077 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-090703a3-5c62-4d75-9bfe-12ddd8a02e7a None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:33.008858 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-090703a3-5c62-4d75-9bfe-12ddd8a02e7a None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:33.043187 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef0daa9c-effa-4096-ad25-c8fe04c688fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:33.043815 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef0daa9c-effa-4096-ad25-c8fe04c688fa None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:33.044095 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef0daa9c-effa-4096-ad25-c8fe04c688fa None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:33.044369 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef0daa9c-effa-4096-ad25-c8fe04c688fa None admin] PATH_INFO: `/v3/projects/fa1ebaa0d49241599f614eb2efe4d8c8` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:33.045480 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef0daa9c-effa-4096-ad25-c8fe04c688fa None admin] RBAC: Authorizing `identity:get_project(project_id=fa1ebaa0d49241599f614eb2efe4d8c8)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:33.046952 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef0daa9c-effa-4096-ad25-c8fe04c688fa None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:33.048568 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 225/453] 10.222.0.90 () {56 vars in 1186 bytes} [Mon Aug 29 21:20:32 2022] GET /identity/v3/projects/fa1ebaa0d49241599f614eb2efe4d8c8 => generated 344 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:33.054203 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee6a7e73-889f-4808-b88a-1d989a609faf None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:33.080951 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:33.153169 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee6a7e73-889f-4808-b88a-1d989a609faf None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:33.186120 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-541a1253-21b8-45ea-9660-b9c571888348 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:33.186743 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-541a1253-21b8-45ea-9660-b9c571888348 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:33.187023 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-541a1253-21b8-45ea-9660-b9c571888348 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:33.187296 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-541a1253-21b8-45ea-9660-b9c571888348 None admin] PATH_INFO: `/v3/groups/1d230e348ec544faac7eeff5f0a65f0a` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:33.188386 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-541a1253-21b8-45ea-9660-b9c571888348 None admin] RBAC: Authorizing `identity:get_group(group_id=1d230e348ec544faac7eeff5f0a65f0a)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:33.189864 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-541a1253-21b8-45ea-9660-b9c571888348 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:33.192008 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 229/454] 10.222.0.90 () {56 vars in 1180 bytes} [Mon Aug 29 21:20:33 2022] GET /identity/v3/groups/1d230e348ec544faac7eeff5f0a65f0a => generated 229 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:33.197923 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef0daa9c-effa-4096-ad25-c8fe04c688fa None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:33.247074 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:33.299078 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef0daa9c-effa-4096-ad25-c8fe04c688fa None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:33.333290 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e6ec5bb-3dd8-4056-8609-e37a0b7cd21b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:33.333920 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e6ec5bb-3dd8-4056-8609-e37a0b7cd21b None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:33.334201 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e6ec5bb-3dd8-4056-8609-e37a0b7cd21b None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:33.334470 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e6ec5bb-3dd8-4056-8609-e37a0b7cd21b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:33.335193 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e6ec5bb-3dd8-4056-8609-e37a0b7cd21b None admin] RBAC: Adding query filter params (group.id=1d230e348ec544faac7eeff5f0a65f0a, role.id=a6e562cdc96c4be691d638990f8403fd, scope.project.id=fa1ebaa0d49241599f614eb2efe4d8c8) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:33.335495 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e6ec5bb-3dd8-4056-8609-e37a0b7cd21b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:33.336914 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e6ec5bb-3dd8-4056-8609-e37a0b7cd21b None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:33.342566 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 226/455] 10.222.0.90 () {56 vars in 1376 bytes} [Mon Aug 29 21:20:33 2022] GET /identity/v3/role_assignments?group.id=1d230e348ec544faac7eeff5f0a65f0a&scope.project.id=fa1ebaa0d49241599f614eb2efe4d8c8&role.id=a6e562cdc96c4be691d638990f8403fd => generated 611 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:35.762664 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39b01c5b-24f5-4cc1-a4a6-c5bf685c86d8 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:35.763182 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39b01c5b-24f5-4cc1-a4a6-c5bf685c86d8 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:35.763182 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39b01c5b-24f5-4cc1-a4a6-c5bf685c86d8 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:35.765265 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 230/456] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:20:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:20:35.770642 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f0b5974-8513-4c76-9569-cd34068aa5f8 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:35.771064 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f0b5974-8513-4c76-9569-cd34068aa5f8 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:35.771233 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f0b5974-8513-4c76-9569-cd34068aa5f8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:35.798847 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-0f0b5974-8513-4c76-9569-cd34068aa5f8 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:20:35.807450 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-0f0b5974-8513-4c76-9569-cd34068aa5f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:20:35.908453 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 227/457] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:20:35 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 21:20:35.998558 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6167d64d-7406-402e-ae4f-e5dd0e0b817d None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:35.998893 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6167d64d-7406-402e-ae4f-e5dd0e0b817d None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:35.999111 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6167d64d-7406-402e-ae4f-e5dd0e0b817d None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:36.000707 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 231/458] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:20:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:20:36.005700 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7abb0529-4588-418b-a9fa-f7c9cc46f2d5 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:36.006022 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7abb0529-4588-418b-a9fa-f7c9cc46f2d5 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:36.006305 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7abb0529-4588-418b-a9fa-f7c9cc46f2d5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:36.030887 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-7abb0529-4588-418b-a9fa-f7c9cc46f2d5 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:20:36.032913 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-304676ce-d1b6-4083-8366-0240016bba19 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:36.033200 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-304676ce-d1b6-4083-8366-0240016bba19 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:36.033479 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-304676ce-d1b6-4083-8366-0240016bba19 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:36.035662 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 232/459] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:20:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:20:36.036624 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-7abb0529-4588-418b-a9fa-f7c9cc46f2d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:20:36.041826 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa6559bf-4fa8-4e69-a967-86ce767721f6 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:36.041826 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa6559bf-4fa8-4e69-a967-86ce767721f6 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:36.041826 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa6559bf-4fa8-4e69-a967-86ce767721f6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:36.061403 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-aa6559bf-4fa8-4e69-a967-86ce767721f6 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:20:36.063142 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-aa6559bf-4fa8-4e69-a967-86ce767721f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:20:36.115901 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:36.176687 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 228/460] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:20:36 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 21:20:36.189461 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 233/461] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:20:36 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 150 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 29 21:20:36.247773 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7abb0529-4588-418b-a9fa-f7c9cc46f2d5 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:36.394620 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7abb0529-4588-418b-a9fa-f7c9cc46f2d5 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:36.444555 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87bc3bcb-f58c-4989-970c-6d98471a772a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:36.445198 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87bc3bcb-f58c-4989-970c-6d98471a772a None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:36.445476 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87bc3bcb-f58c-4989-970c-6d98471a772a None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:36.445750 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87bc3bcb-f58c-4989-970c-6d98471a772a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:36.461520 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87bc3bcb-f58c-4989-970c-6d98471a772a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:36.463028 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87bc3bcb-f58c-4989-970c-6d98471a772a None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:36.481648 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-87bc3bcb-f58c-4989-970c-6d98471a772a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:20:36.481648 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:36.481648 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:36.481648 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:36.481648 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:36.481648 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:36.481648 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:36.481648 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:36.481648 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:36.481648 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:36.481648 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:36.481648 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:36.481648 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 21:20:36.481648 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 21:20:36.481648 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 21:20:36.481648 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 21:20:36.482833 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:36.482833 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:36.482833 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:36.482833 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:36.482833 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:36.482833 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:36.482833 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:36.482833 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:36.482833 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:36.482833 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:36.482833 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:36.482833 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:36.482833 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:36.482833 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:36.482833 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:36.482833 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:36.482833 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 21:20:36.483990 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 21:20:36.483990 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:20:36.483990 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:20:36.483990 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 229/462] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:20:36 2022] GET /identity/v3/domains/Default => generated 87 bytes in 236 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:20:36.488759 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa6559bf-4fa8-4e69-a967-86ce767721f6 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:36.488996 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87bc3bcb-f58c-4989-970c-6d98471a772a None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:36.522040 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:36.529268 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:36.611622 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa6559bf-4fa8-4e69-a967-86ce767721f6 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:36.621014 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87bc3bcb-f58c-4989-970c-6d98471a772a None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:36.653646 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f27e36e9-9250-4dae-966d-5d4ed69af15c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:36.654365 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f27e36e9-9250-4dae-966d-5d4ed69af15c None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:36.655023 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f27e36e9-9250-4dae-966d-5d4ed69af15c None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:36.655023 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f27e36e9-9250-4dae-966d-5d4ed69af15c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:36.657852 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b573da47-7203-4652-bbbe-fcad8478ef56 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:36.658618 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b573da47-7203-4652-bbbe-fcad8478ef56 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:36.659036 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b573da47-7203-4652-bbbe-fcad8478ef56 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:36.659966 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b573da47-7203-4652-bbbe-fcad8478ef56 None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:36.660842 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b573da47-7203-4652-bbbe-fcad8478ef56 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:36.661299 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b573da47-7203-4652-bbbe-fcad8478ef56 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:36.662955 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b573da47-7203-4652-bbbe-fcad8478ef56 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:36.666228 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f27e36e9-9250-4dae-966d-5d4ed69af15c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:36.667684 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f27e36e9-9250-4dae-966d-5d4ed69af15c None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:36.676686 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 230/463] 10.222.0.90 () {56 vars in 1109 bytes} [Mon Aug 29 21:20:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:20:36.677706 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b573da47-7203-4652-bbbe-fcad8478ef56 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:36.686479 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-f27e36e9-9250-4dae-966d-5d4ed69af15c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:20:36.686479 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:36.686479 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:36.686479 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:36.686479 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:36.686479 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:36.686479 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:36.686479 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:36.686479 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:36.686479 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:36.686479 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:36.686479 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:36.686479 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 21:20:36.686479 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 21:20:36.686479 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 21:20:36.686479 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 21:20:36.687421 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:36.687421 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:36.687421 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:36.687421 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:36.687421 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:36.687421 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:36.687421 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:36.687421 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:36.687421 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:36.687421 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:36.687421 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:36.687421 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:36.687421 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:36.687421 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:36.687421 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:36.687421 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:36.687421 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 21:20:36.688292 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 21:20:36.688292 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:20:36.688292 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:20:36.688292 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 234/464] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:20:36 2022] GET /identity/v3/domains/Default => generated 87 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:20:36.689136 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a8bf78b-efbc-4417-a4bf-88f1c0fd3fcd None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:36.689420 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a8bf78b-efbc-4417-a4bf-88f1c0fd3fcd None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:36.689694 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a8bf78b-efbc-4417-a4bf-88f1c0fd3fcd None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:36.695136 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 235/465] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:20:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:20:36.695136 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a8bf78b-efbc-4417-a4bf-88f1c0fd3fcd None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:36.780597 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:36.782589 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:36.800595 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b573da47-7203-4652-bbbe-fcad8478ef56 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:36.805604 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a8bf78b-efbc-4417-a4bf-88f1c0fd3fcd None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:36.837408 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4b13be4-c1b7-430a-88d6-5ac57ed5458f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:36.838398 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4b13be4-c1b7-430a-88d6-5ac57ed5458f None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:36.838898 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4b13be4-c1b7-430a-88d6-5ac57ed5458f None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:36.839380 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4b13be4-c1b7-430a-88d6-5ac57ed5458f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:36.840266 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd254600-10dd-4f9a-83a7-8db442185d81 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:36.840869 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd254600-10dd-4f9a-83a7-8db442185d81 None admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:36.841157 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd254600-10dd-4f9a-83a7-8db442185d81 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:36.841434 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd254600-10dd-4f9a-83a7-8db442185d81 None admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:36.842132 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd254600-10dd-4f9a-83a7-8db442185d81 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:36.843596 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd254600-10dd-4f9a-83a7-8db442185d81 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:36.860952 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4b13be4-c1b7-430a-88d6-5ac57ed5458f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:36.862713 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4b13be4-c1b7-430a-88d6-5ac57ed5458f None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:36.869835 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 236/466] 10.222.0.90 () {60 vars in 1132 bytes} [Mon Aug 29 21:20:36 2022] POST /identity/v3/users => generated 246 bytes in 178 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:20:36.882402 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd254600-10dd-4f9a-83a7-8db442185d81 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:36.891544 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-f4b13be4-c1b7-430a-88d6-5ac57ed5458f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:20:36.891544 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:36.891544 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:36.891544 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:36.891544 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:36.891544 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:36.891544 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:36.891544 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:36.891544 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:36.891544 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:36.891544 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:36.891544 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:36.891544 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 21:20:36.891544 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 21:20:36.891544 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 21:20:36.891544 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 21:20:36.892611 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:36.892611 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:36.892611 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:36.892611 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:36.892611 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:36.892611 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:36.892611 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:36.892611 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:36.892611 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:36.892611 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:36.892611 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:36.892611 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:36.892611 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:36.892611 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:36.892611 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:36.892611 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:36.892611 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 21:20:36.893469 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 21:20:36.893469 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:20:36.893469 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:20:36.893469 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 231/467] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:20:36 2022] GET /identity/v3/domains/Default => generated 87 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:20:36.898215 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aec42be4-6795-4409-974d-2febd600e24f None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:36.898800 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aec42be4-6795-4409-974d-2febd600e24f None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:36.899320 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aec42be4-6795-4409-974d-2febd600e24f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:36.919760 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-aec42be4-6795-4409-974d-2febd600e24f None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:20:36.921250 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-aec42be4-6795-4409-974d-2febd600e24f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:20:37.006213 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 232/468] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:20:36 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 21:20:37.007454 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aec42be4-6795-4409-974d-2febd600e24f None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:37.008795 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd254600-10dd-4f9a-83a7-8db442185d81 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:37.048631 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0abfdce9-0033-4e59-953d-0a469b2ae551 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:37.049252 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0abfdce9-0033-4e59-953d-0a469b2ae551 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:37.049535 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0abfdce9-0033-4e59-953d-0a469b2ae551 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:37.049825 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0abfdce9-0033-4e59-953d-0a469b2ae551 None admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:37.050525 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0abfdce9-0033-4e59-953d-0a469b2ae551 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:37.050942 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0abfdce9-0033-4e59-953d-0a469b2ae551 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:37.052425 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0abfdce9-0033-4e59-953d-0a469b2ae551 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:37.054845 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:37.071392 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 237/469] 10.222.0.90 () {56 vars in 1109 bytes} [Mon Aug 29 21:20:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:37.077380 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0abfdce9-0033-4e59-953d-0a469b2ae551 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:37.112641 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:37.157884 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aec42be4-6795-4409-974d-2febd600e24f None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:37.205860 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0abfdce9-0033-4e59-953d-0a469b2ae551 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:37.208776 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55ef3061-cb68-468a-9263-7a6d230f39af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:37.210304 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ef3061-cb68-468a-9263-7a6d230f39af None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:37.210650 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ef3061-cb68-468a-9263-7a6d230f39af None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:37.210930 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ef3061-cb68-468a-9263-7a6d230f39af None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:37.211649 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55ef3061-cb68-468a-9263-7a6d230f39af None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:37.211958 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55ef3061-cb68-468a-9263-7a6d230f39af None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:37.213353 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55ef3061-cb68-468a-9263-7a6d230f39af None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:37.230302 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 233/470] 10.222.0.90 () {56 vars in 1109 bytes} [Mon Aug 29 21:20:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 223 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:37.231548 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea93cb44-d2c0-4b91-8fab-15666f781d87 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:37.231837 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea93cb44-d2c0-4b91-8fab-15666f781d87 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:37.232391 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea93cb44-d2c0-4b91-8fab-15666f781d87 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:37.233564 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 234/471] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:20:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:20:37.238401 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea93cb44-d2c0-4b91-8fab-15666f781d87 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:37.250120 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9354389-d412-461d-9302-0f51aef47200 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:37.250781 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9354389-d412-461d-9302-0f51aef47200 None admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:37.251063 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9354389-d412-461d-9302-0f51aef47200 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:37.251338 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9354389-d412-461d-9302-0f51aef47200 None admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:37.252001 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9354389-d412-461d-9302-0f51aef47200 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:37.254000 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9354389-d412-461d-9302-0f51aef47200 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:37.279603 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 238/472] 10.222.0.90 () {60 vars in 1132 bytes} [Mon Aug 29 21:20:37 2022] POST /identity/v3/users => generated 248 bytes in 203 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 21:20:37.281126 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89e701d5-2e56-4ccf-9982-76cab724b5f7 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:37.281414 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89e701d5-2e56-4ccf-9982-76cab724b5f7 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:37.281747 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89e701d5-2e56-4ccf-9982-76cab724b5f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:37.323026 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-89e701d5-2e56-4ccf-9982-76cab724b5f7 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:20:37.324267 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-89e701d5-2e56-4ccf-9982-76cab724b5f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:20:37.390202 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:37.399233 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea93cb44-d2c0-4b91-8fab-15666f781d87 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:37.405527 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 239/473] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:20:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 21:20:37.406640 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89e701d5-2e56-4ccf-9982-76cab724b5f7 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:37.445073 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:37.455281 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-278d21ae-b454-476e-8e1e-9d9d5eb5baa4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:37.456058 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-278d21ae-b454-476e-8e1e-9d9d5eb5baa4 None admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:37.456282 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-278d21ae-b454-476e-8e1e-9d9d5eb5baa4 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:37.458521 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-278d21ae-b454-476e-8e1e-9d9d5eb5baa4 None admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:37.460105 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-278d21ae-b454-476e-8e1e-9d9d5eb5baa4 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:37.461656 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-278d21ae-b454-476e-8e1e-9d9d5eb5baa4 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:37.490887 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 235/474] 10.222.0.90 () {60 vars in 1132 bytes} [Mon Aug 29 21:20:37 2022] POST /identity/v3/users => generated 248 bytes in 253 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:20:37.538004 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89e701d5-2e56-4ccf-9982-76cab724b5f7 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:37.584258 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d805b6f-0fca-4512-b561-8a77a5ef8334 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:37.584919 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d805b6f-0fca-4512-b561-8a77a5ef8334 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:37.585197 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d805b6f-0fca-4512-b561-8a77a5ef8334 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:37.585466 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d805b6f-0fca-4512-b561-8a77a5ef8334 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:37.600556 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d805b6f-0fca-4512-b561-8a77a5ef8334 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:37.602062 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d805b6f-0fca-4512-b561-8a77a5ef8334 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:37.623038 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-2d805b6f-0fca-4512-b561-8a77a5ef8334 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:20:37.623038 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:37.623038 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:37.623038 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:37.623038 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:37.623038 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:37.623038 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:37.623038 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:37.623038 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:37.623038 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:37.623038 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:37.623038 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:37.623038 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 21:20:37.623038 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 21:20:37.623038 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 21:20:37.623038 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 21:20:37.625224 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:37.625224 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:37.625224 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:37.625224 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:37.625224 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:37.625224 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:37.625224 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:37.625224 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:37.625224 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:37.625224 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:37.625224 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:37.625224 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:37.625224 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:37.625224 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:37.625224 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:37.625224 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:37.625224 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 21:20:37.627460 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 21:20:37.627460 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:20:37.627460 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:20:37.630354 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 240/475] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:20:37 2022] GET /identity/v3/domains/Default => generated 87 bytes in 224 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:20:37.632032 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-278d21ae-b454-476e-8e1e-9d9d5eb5baa4 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:37.702008 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d805b6f-0fca-4512-b561-8a77a5ef8334 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:37.747502 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:37.767702 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-278d21ae-b454-476e-8e1e-9d9d5eb5baa4 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:37.806252 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1e3f517-f1f5-4d4c-9733-cc5617559028 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:37.807649 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1e3f517-f1f5-4d4c-9733-cc5617559028 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:37.808151 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1e3f517-f1f5-4d4c-9733-cc5617559028 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:37.808617 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1e3f517-f1f5-4d4c-9733-cc5617559028 None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:37.809460 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1e3f517-f1f5-4d4c-9733-cc5617559028 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:37.810037 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1e3f517-f1f5-4d4c-9733-cc5617559028 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:37.811639 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1e3f517-f1f5-4d4c-9733-cc5617559028 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:37.825339 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 236/476] 10.222.0.90 () {56 vars in 1109 bytes} [Mon Aug 29 21:20:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:37.830682 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1e3f517-f1f5-4d4c-9733-cc5617559028 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:37.852437 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d805b6f-0fca-4512-b561-8a77a5ef8334 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:37.886181 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:37.920444 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9fa0b83-66ba-4856-9a40-ce6f623e0644 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:37.921114 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9fa0b83-66ba-4856-9a40-ce6f623e0644 None admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:37.921389 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9fa0b83-66ba-4856-9a40-ce6f623e0644 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:37.921654 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9fa0b83-66ba-4856-9a40-ce6f623e0644 None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:37.922414 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9fa0b83-66ba-4856-9a40-ce6f623e0644 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:37.923771 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9fa0b83-66ba-4856-9a40-ce6f623e0644 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:37.935583 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.sql.core [None req-a9fa0b83-66ba-4856-9a40-ce6f623e0644 None admin] Conflict role: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Aug 29 21:20:37.935583 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Aug 29 21:20:37.935583 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [parameters: {'id': 'c0526dba5f23456b9aeb4de88cbc5695', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Aug 29 21:20:37.935583 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81434) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Aug 29 21:20:37.960921 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-a9fa0b83-66ba-4856-9a40-ce6f623e0644 None admin] Conflict occurred attempting to store role - Duplicate entry found with name anotherrole.: keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Aug 29 21:20:37.960921 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:37.960921 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Aug 29 21:20:37.960921 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application self.dialect.do_execute( Aug 29 21:20:37.960921 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Aug 29 21:20:37.960921 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Aug 29 21:20:37.960921 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Aug 29 21:20:37.960921 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application result = self._query(query) Aug 29 21:20:37.960921 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Aug 29 21:20:37.960921 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application conn.query(q) Aug 29 21:20:37.960921 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Aug 29 21:20:37.960921 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Aug 29 21:20:37.960921 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Aug 29 21:20:37.960921 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application result.read() Aug 29 21:20:37.960921 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Aug 29 21:20:37.961863 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Aug 29 21:20:37.961863 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Aug 29 21:20:37.961863 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application packet.raise_for_error() Aug 29 21:20:37.961863 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Aug 29 21:20:37.961863 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Aug 29 21:20:37.961863 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Aug 29 21:20:37.961863 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Aug 29 21:20:37.961863 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Aug 29 21:20:37.961863 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:20:37.961863 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Aug 29 21:20:37.961863 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:20:37.961863 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:37.961863 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Aug 29 21:20:37.961863 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return method(*args, **kwargs) Aug 29 21:20:37.961863 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 33, in create_role Aug 29 21:20:37.961863 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return ref.to_dict() Aug 29 21:20:37.961863 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Aug 29 21:20:37.961863 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application next(self.gen) Aug 29 21:20:37.962809 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Aug 29 21:20:37.962809 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application yield resource Aug 29 21:20:37.962809 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Aug 29 21:20:37.962809 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application next(self.gen) Aug 29 21:20:37.962809 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Aug 29 21:20:37.962809 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application self.session.rollback() Aug 29 21:20:37.962809 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Aug 29 21:20:37.962809 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application self.force_reraise() Aug 29 21:20:37.962809 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Aug 29 21:20:37.962809 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise self.value Aug 29 21:20:37.962809 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Aug 29 21:20:37.962809 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Aug 29 21:20:37.962809 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Aug 29 21:20:37.962809 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application session.commit() Aug 29 21:20:37.962809 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Aug 29 21:20:37.962809 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Aug 29 21:20:37.962809 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Aug 29 21:20:37.963682 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application self._prepare_impl() Aug 29 21:20:37.963682 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Aug 29 21:20:37.963682 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application self.session.flush() Aug 29 21:20:37.963682 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Aug 29 21:20:37.963682 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application self._flush(objects) Aug 29 21:20:37.963682 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Aug 29 21:20:37.963682 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Aug 29 21:20:37.963682 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Aug 29 21:20:37.963682 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application compat.raise_( Aug 29 21:20:37.963682 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Aug 29 21:20:37.963682 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception Aug 29 21:20:37.963682 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Aug 29 21:20:37.963682 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application flush_context.execute() Aug 29 21:20:37.963682 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Aug 29 21:20:37.963682 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rec.execute(self) Aug 29 21:20:37.963682 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Aug 29 21:20:37.963682 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Aug 29 21:20:37.964565 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Aug 29 21:20:37.964565 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application _emit_insert_statements( Aug 29 21:20:37.964565 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Aug 29 21:20:37.964565 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application c = connection._execute_20( Aug 29 21:20:37.964565 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Aug 29 21:20:37.964565 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Aug 29 21:20:37.964565 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Aug 29 21:20:37.964565 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Aug 29 21:20:37.964565 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Aug 29 21:20:37.964565 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ret = self._execute_context( Aug 29 21:20:37.964565 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Aug 29 21:20:37.964565 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Aug 29 21:20:37.964565 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Aug 29 21:20:37.964565 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Aug 29 21:20:37.964565 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Aug 29 21:20:37.964565 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception Aug 29 21:20:37.965389 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Aug 29 21:20:37.965389 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application self.dialect.do_execute( Aug 29 21:20:37.965389 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Aug 29 21:20:37.965389 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Aug 29 21:20:37.965389 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Aug 29 21:20:37.965389 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application result = self._query(query) Aug 29 21:20:37.965389 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Aug 29 21:20:37.965389 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application conn.query(q) Aug 29 21:20:37.965389 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Aug 29 21:20:37.965389 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Aug 29 21:20:37.965389 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Aug 29 21:20:37.965389 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application result.read() Aug 29 21:20:37.965389 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Aug 29 21:20:37.965389 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Aug 29 21:20:37.965389 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Aug 29 21:20:37.965389 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application packet.raise_for_error() Aug 29 21:20:37.966215 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Aug 29 21:20:37.966215 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Aug 29 21:20:37.966215 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Aug 29 21:20:37.966215 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Aug 29 21:20:37.966215 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Aug 29 21:20:37.966215 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Aug 29 21:20:37.966215 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [parameters: {'id': 'c0526dba5f23456b9aeb4de88cbc5695', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Aug 29 21:20:37.966215 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Aug 29 21:20:37.966215 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:20:37.966215 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 29 21:20:37.966215 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:20:37.966215 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:37.966215 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:37.966215 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:37.966215 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:37.966215 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:37.967087 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:37.967087 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:37.967087 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:37.967087 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:37.967087 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:37.967087 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:37.967087 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 115, in post Aug 29 21:20:37.967087 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ref = PROVIDERS.role_api.create_role( Aug 29 21:20:37.967087 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:37.967087 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:37.967087 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1298, in create_role Aug 29 21:20:37.967087 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ret = self.driver.create_role(role_id, role) Aug 29 21:20:37.967087 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Aug 29 21:20:37.967087 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Aug 29 21:20:37.967087 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Aug 29 21:20:37.967087 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:20:37.970855 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 241/477] 10.222.0.90 () {60 vars in 1131 bytes} [Mon Aug 29 21:20:37 2022] POST /identity/v3/roles => generated 144 bytes in 269 msecs (HTTP/1.1 409) 5 headers in 183 bytes (2 switches on core 0) Aug 29 21:20:37.978065 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9fa0b83-66ba-4856-9a40-ce6f623e0644 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:38.003066 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1e3f517-f1f5-4d4c-9733-cc5617559028 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:38.083786 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53c525cc-d7d4-4647-b15a-7895f13caecc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:38.084925 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53c525cc-d7d4-4647-b15a-7895f13caecc None admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:38.086259 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53c525cc-d7d4-4647-b15a-7895f13caecc None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:38.086539 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53c525cc-d7d4-4647-b15a-7895f13caecc None admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:38.087211 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53c525cc-d7d4-4647-b15a-7895f13caecc None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:38.088649 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53c525cc-d7d4-4647-b15a-7895f13caecc None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:38.124447 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 237/478] 10.222.0.90 () {60 vars in 1132 bytes} [Mon Aug 29 21:20:37 2022] POST /identity/v3/users => generated 249 bytes in 294 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 21:20:38.164594 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9fa0b83-66ba-4856-9a40-ce6f623e0644 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:38.174315 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:38.214427 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc1ebd5e-c6f2-44f3-8bba-df70f8fc0dc1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:38.215088 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc1ebd5e-c6f2-44f3-8bba-df70f8fc0dc1 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:38.215367 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc1ebd5e-c6f2-44f3-8bba-df70f8fc0dc1 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:38.215705 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc1ebd5e-c6f2-44f3-8bba-df70f8fc0dc1 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:38.228569 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc1ebd5e-c6f2-44f3-8bba-df70f8fc0dc1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 21:20:38.234308 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-fc1ebd5e-c6f2-44f3-8bba-df70f8fc0dc1 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 29 21:20:38.234308 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:38.234308 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:38.234308 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:38.234308 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:38.234308 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:38.234308 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:38.234308 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:38.234308 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:38.234308 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:38.234308 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:38.234308 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:38.234308 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 21:20:38.234308 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 21:20:38.234308 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 21:20:38.234308 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 21:20:38.235222 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 21:20:38.235222 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 21:20:38.235222 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 21:20:38.235222 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 21:20:38.235222 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:38.235222 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:38.235222 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:38.235222 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:38.235222 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:38.235222 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:38.235222 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:38.235222 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:38.235222 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:38.235222 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:38.235222 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:38.235222 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:38.236010 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:38.236010 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:38.236010 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:38.236010 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:38.236010 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 21:20:38.236010 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 21:20:38.236010 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 21:20:38.236010 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 21:20:38.236010 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 21:20:38.236010 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 21:20:38.236010 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 29 21:20:38.236010 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:20:38.236010 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 242/479] 10.222.0.90 () {56 vars in 1114 bytes} [Mon Aug 29 21:20:37 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 258 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:20:38.241482 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53c525cc-d7d4-4647-b15a-7895f13caecc None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:38.375784 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:38.379787 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53c525cc-d7d4-4647-b15a-7895f13caecc None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:38.426966 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1534eae-34c9-4a7a-b4f3-1e523a29e38f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:38.427587 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1534eae-34c9-4a7a-b4f3-1e523a29e38f None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:38.427870 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1534eae-34c9-4a7a-b4f3-1e523a29e38f None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:38.428142 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1534eae-34c9-4a7a-b4f3-1e523a29e38f None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:38.428821 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1534eae-34c9-4a7a-b4f3-1e523a29e38f None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:38.429124 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1534eae-34c9-4a7a-b4f3-1e523a29e38f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:38.430536 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1534eae-34c9-4a7a-b4f3-1e523a29e38f None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:38.441440 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 238/480] 10.222.0.90 () {56 vars in 1111 bytes} [Mon Aug 29 21:20:38 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:39.323954 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f02be67b-5039-4cf4-ac95-fbf9f40cf60c None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:39.324974 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f02be67b-5039-4cf4-ac95-fbf9f40cf60c None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:39.325456 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f02be67b-5039-4cf4-ac95-fbf9f40cf60c None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:39.327308 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 243/481] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:20:39 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:20:39.333655 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f509230b-7713-42e6-8d49-4213f3e3c630 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:39.334134 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f509230b-7713-42e6-8d49-4213f3e3c630 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:39.334316 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f509230b-7713-42e6-8d49-4213f3e3c630 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:39.358834 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-f509230b-7713-42e6-8d49-4213f3e3c630 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:20:39.360370 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-f509230b-7713-42e6-8d49-4213f3e3c630 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:20:39.436150 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 239/482] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:20:39 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 21:20:39.766430 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08383447-c3da-4928-8f16-5f9f4aa81aa6 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:39.766718 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08383447-c3da-4928-8f16-5f9f4aa81aa6 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:39.766913 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08383447-c3da-4928-8f16-5f9f4aa81aa6 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:39.768487 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 244/483] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:20:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:20:39.775498 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76860d5a-9619-47b3-b11c-792d04753214 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:39.775794 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76860d5a-9619-47b3-b11c-792d04753214 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:39.776077 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76860d5a-9619-47b3-b11c-792d04753214 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:39.795911 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-76860d5a-9619-47b3-b11c-792d04753214 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:20:39.797172 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-76860d5a-9619-47b3-b11c-792d04753214 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:20:39.843841 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08383447-c3da-4928-8f16-5f9f4aa81aa6 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:39.909427 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 240/484] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:20:39 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 21:20:39.934297 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0200830-f76d-4bed-97d7-e447fb2147c4 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:39.934610 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0200830-f76d-4bed-97d7-e447fb2147c4 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:39.934923 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0200830-f76d-4bed-97d7-e447fb2147c4 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:39.936630 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 241/485] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:20:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 29 21:20:39.951222 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-741c21f5-43b3-4c07-8e06-e0f698ee2b25 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:39.951541 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-741c21f5-43b3-4c07-8e06-e0f698ee2b25 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:39.951809 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-741c21f5-43b3-4c07-8e06-e0f698ee2b25 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:39.980894 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-741c21f5-43b3-4c07-8e06-e0f698ee2b25 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:20:39.983810 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-741c21f5-43b3-4c07-8e06-e0f698ee2b25 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:20:40.011352 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:40.039122 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08383447-c3da-4928-8f16-5f9f4aa81aa6 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:40.099415 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 242/486] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:20:39 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 29 21:20:40.106716 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24e871c9-9ae1-413d-b7b8-cb3cf49ec813 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:40.107615 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24e871c9-9ae1-413d-b7b8-cb3cf49ec813 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:40.108107 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24e871c9-9ae1-413d-b7b8-cb3cf49ec813 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:40.108579 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24e871c9-9ae1-413d-b7b8-cb3cf49ec813 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:40.126605 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24e871c9-9ae1-413d-b7b8-cb3cf49ec813 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 21:20:40.131735 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-24e871c9-9ae1-413d-b7b8-cb3cf49ec813 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 29 21:20:40.131735 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:40.131735 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:40.131735 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:40.131735 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:40.131735 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:40.131735 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:40.131735 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:40.131735 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:40.131735 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:40.131735 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:40.131735 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:40.131735 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 21:20:40.131735 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 21:20:40.131735 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 21:20:40.131735 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 21:20:40.132632 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 21:20:40.132632 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 21:20:40.132632 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 21:20:40.132632 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 21:20:40.132632 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:40.132632 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:40.132632 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:40.132632 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:40.132632 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:40.132632 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:40.132632 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:40.132632 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:40.132632 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:40.132632 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:40.132632 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:40.132632 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:40.133459 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:40.133459 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:40.133459 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:40.133459 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:40.133459 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 21:20:40.133459 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 21:20:40.133459 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 21:20:40.133459 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 21:20:40.133459 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 21:20:40.133459 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 21:20:40.133459 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 29 21:20:40.133459 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:20:40.136935 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 245/487] 10.222.0.90 () {56 vars in 1102 bytes} [Mon Aug 29 21:20:39 2022] GET /identity/v3/roles/service => generated 85 bytes in 292 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:20:40.142036 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-741c21f5-43b3-4c07-8e06-e0f698ee2b25 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:40.264227 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24e871c9-9ae1-413d-b7b8-cb3cf49ec813 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:40.269121 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-741c21f5-43b3-4c07-8e06-e0f698ee2b25 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:40.310239 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:40.324390 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b6b3f53-f96d-4afb-9411-a867c3ee2839 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:40.325969 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b6b3f53-f96d-4afb-9411-a867c3ee2839 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:40.326451 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b6b3f53-f96d-4afb-9411-a867c3ee2839 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:40.326921 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b6b3f53-f96d-4afb-9411-a867c3ee2839 None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:40.327809 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b6b3f53-f96d-4afb-9411-a867c3ee2839 None admin] RBAC: Adding query filter params (name=service) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:40.328311 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b6b3f53-f96d-4afb-9411-a867c3ee2839 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:40.329913 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b6b3f53-f96d-4afb-9411-a867c3ee2839 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:40.343539 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 243/488] 10.222.0.90 () {56 vars in 1103 bytes} [Mon Aug 29 21:20:40 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:40.351029 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b6b3f53-f96d-4afb-9411-a867c3ee2839 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:40.379874 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:40.469998 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b6b3f53-f96d-4afb-9411-a867c3ee2839 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:40.506266 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77d99b35-bb55-4fb8-9b41-25e505d04849 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:40.506767 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24e871c9-9ae1-413d-b7b8-cb3cf49ec813 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:40.507818 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77d99b35-bb55-4fb8-9b41-25e505d04849 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:40.508297 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77d99b35-bb55-4fb8-9b41-25e505d04849 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:40.508792 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77d99b35-bb55-4fb8-9b41-25e505d04849 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:40.520890 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77d99b35-bb55-4fb8-9b41-25e505d04849 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:40.522732 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77d99b35-bb55-4fb8-9b41-25e505d04849 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:40.539815 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-77d99b35-bb55-4fb8-9b41-25e505d04849 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:20:40.539815 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:40.539815 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:40.539815 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:40.539815 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:40.539815 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:40.539815 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:40.539815 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:40.539815 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:40.539815 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:40.539815 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:40.539815 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:40.539815 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 21:20:40.539815 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 21:20:40.539815 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 21:20:40.539815 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 21:20:40.540794 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:40.540794 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:40.540794 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:40.540794 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:40.540794 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:40.540794 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:40.540794 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:40.540794 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:40.540794 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:40.540794 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:40.540794 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:40.540794 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:40.540794 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:40.540794 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:40.540794 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:40.540794 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:40.540794 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 21:20:40.541736 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 21:20:40.541736 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:20:40.541736 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:20:40.548426 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 244/489] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:20:40 2022] GET /identity/v3/domains/Default => generated 87 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 29 21:20:40.549901 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77d99b35-bb55-4fb8-9b41-25e505d04849 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:40.582574 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0cfed20-2124-4a15-b3ae-ab12717b0d44 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:40.583416 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0cfed20-2124-4a15-b3ae-ab12717b0d44 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:40.583545 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0cfed20-2124-4a15-b3ae-ab12717b0d44 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:40.583783 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0cfed20-2124-4a15-b3ae-ab12717b0d44 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:40.594975 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0cfed20-2124-4a15-b3ae-ab12717b0d44 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 21:20:40.601893 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-d0cfed20-2124-4a15-b3ae-ab12717b0d44 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 29 21:20:40.601893 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:40.601893 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:40.601893 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:40.601893 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:40.601893 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:40.601893 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:40.601893 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:40.601893 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:40.601893 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:40.601893 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:40.601893 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:40.601893 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 21:20:40.601893 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 21:20:40.601893 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 21:20:40.601893 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 21:20:40.602812 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 21:20:40.602812 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 21:20:40.602812 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 21:20:40.602812 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 21:20:40.602812 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:40.602812 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:40.602812 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:40.602812 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:40.602812 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:40.602812 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:40.602812 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:40.602812 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:40.602812 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:40.602812 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:40.602812 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:40.602812 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:40.603617 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:40.603617 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:40.603617 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:40.603617 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:40.603617 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 21:20:40.603617 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 21:20:40.603617 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 21:20:40.603617 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 21:20:40.603617 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 21:20:40.603617 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 21:20:40.603617 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 29 21:20:40.603617 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:20:40.607955 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 246/490] 10.222.0.90 () {56 vars in 1102 bytes} [Mon Aug 29 21:20:40 2022] GET /identity/v3/roles/service => generated 85 bytes in 339 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 29 21:20:40.608828 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0cfed20-2124-4a15-b3ae-ab12717b0d44 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:40.624282 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:40.663961 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77d99b35-bb55-4fb8-9b41-25e505d04849 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:40.699255 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-add839c8-26e8-4e8d-aebc-ee633921476e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:40.700304 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-add839c8-26e8-4e8d-aebc-ee633921476e None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:40.700788 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-add839c8-26e8-4e8d-aebc-ee633921476e None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:40.701262 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-add839c8-26e8-4e8d-aebc-ee633921476e None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:40.701687 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:40.702364 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-add839c8-26e8-4e8d-aebc-ee633921476e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:40.702881 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-add839c8-26e8-4e8d-aebc-ee633921476e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:40.704494 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-add839c8-26e8-4e8d-aebc-ee633921476e None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:40.722869 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 245/491] 10.222.0.90 () {56 vars in 1109 bytes} [Mon Aug 29 21:20:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:20:40.724160 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-add839c8-26e8-4e8d-aebc-ee633921476e None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:40.730645 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0cfed20-2124-4a15-b3ae-ab12717b0d44 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:40.773227 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69ed1797-8afa-4b82-8fe6-8e83bd77b0fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:40.773910 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ed1797-8afa-4b82-8fe6-8e83bd77b0fc None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:40.774194 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ed1797-8afa-4b82-8fe6-8e83bd77b0fc None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:40.774469 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ed1797-8afa-4b82-8fe6-8e83bd77b0fc None admin] PATH_INFO: `/v3/roles/service` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:40.786864 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69ed1797-8afa-4b82-8fe6-8e83bd77b0fc None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 21:20:40.793176 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-69ed1797-8afa-4b82-8fe6-8e83bd77b0fc None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 29 21:20:40.793176 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:40.793176 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:40.793176 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:40.793176 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:40.793176 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:40.793176 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:40.793176 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:40.793176 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:40.793176 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:40.793176 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:40.793176 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:40.793176 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 21:20:40.793176 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 21:20:40.793176 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 21:20:40.793176 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 21:20:40.794453 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 21:20:40.794453 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 21:20:40.794453 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 21:20:40.794453 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 21:20:40.794453 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:40.794453 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:40.794453 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:40.794453 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:40.794453 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:40.794453 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:40.794453 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:40.794453 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:40.794453 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:40.794453 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:40.794453 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:40.794453 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:40.795316 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:40.795316 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:40.795316 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:40.795316 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:40.795316 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 21:20:40.795316 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 21:20:40.795316 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 21:20:40.795316 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 21:20:40.795316 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 21:20:40.795316 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 21:20:40.795316 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 29 21:20:40.795316 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:20:40.795316 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 247/492] 10.222.0.90 () {56 vars in 1102 bytes} [Mon Aug 29 21:20:40 2022] GET /identity/v3/roles/service => generated 85 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:20:40.795316 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69ed1797-8afa-4b82-8fe6-8e83bd77b0fc None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:40.834613 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-add839c8-26e8-4e8d-aebc-ee633921476e None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:40.861422 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:40.869696 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fe36b59-2cd7-4d34-80fc-d6cb055f7c76 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:40.870333 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fe36b59-2cd7-4d34-80fc-d6cb055f7c76 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:40.870610 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fe36b59-2cd7-4d34-80fc-d6cb055f7c76 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:40.870880 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fe36b59-2cd7-4d34-80fc-d6cb055f7c76 None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:40.871534 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fe36b59-2cd7-4d34-80fc-d6cb055f7c76 None admin] RBAC: Adding query filter params (name=service) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:40.871831 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fe36b59-2cd7-4d34-80fc-d6cb055f7c76 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:40.873180 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fe36b59-2cd7-4d34-80fc-d6cb055f7c76 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:40.881778 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 246/493] 10.222.0.90 () {56 vars in 1103 bytes} [Mon Aug 29 21:20:40 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:40.882963 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-172d2c7f-dde5-4533-9469-f715d45f0c17 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:40.883242 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-172d2c7f-dde5-4533-9469-f715d45f0c17 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:40.883511 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-172d2c7f-dde5-4533-9469-f715d45f0c17 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:40.884842 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 247/494] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:20:40 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:20:40.889114 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-172d2c7f-dde5-4533-9469-f715d45f0c17 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:40.904085 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69ed1797-8afa-4b82-8fe6-8e83bd77b0fc None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:40.937903 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdd900d9-544a-42fe-9af0-2bd66beb8039 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:40.938534 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd900d9-544a-42fe-9af0-2bd66beb8039 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:40.938814 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd900d9-544a-42fe-9af0-2bd66beb8039 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:40.939085 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd900d9-544a-42fe-9af0-2bd66beb8039 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:40.942580 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:40.948434 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdd900d9-544a-42fe-9af0-2bd66beb8039 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:40.950024 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdd900d9-544a-42fe-9af0-2bd66beb8039 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:40.963116 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-bdd900d9-544a-42fe-9af0-2bd66beb8039 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 29 21:20:40.963116 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:40.963116 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:40.963116 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:40.963116 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:40.963116 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:40.963116 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:40.963116 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:40.963116 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:40.963116 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:40.963116 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:40.963116 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:40.963116 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 29 21:20:40.963116 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 29 21:20:40.963116 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 29 21:20:40.963116 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 29 21:20:40.964079 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:40.964079 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:40.964079 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 29 21:20:40.964079 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 21:20:40.964079 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 29 21:20:40.964079 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 21:20:40.964079 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:40.964079 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:40.964079 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:40.964079 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:40.964079 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:40.964079 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:40.964079 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:40.964079 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:40.964079 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:40.964079 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:40.964079 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:40.964079 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:40.964996 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:40.964996 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:40.964996 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 29 21:20:40.964996 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 29 21:20:40.964996 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 29 21:20:40.964996 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 29 21:20:40.964996 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 29 21:20:40.964996 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 29 21:20:40.964996 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 29 21:20:40.964996 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:20:40.964996 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 248/495] 10.222.0.90 () {56 vars in 1093 bytes} [Mon Aug 29 21:20:40 2022] GET /identity/v3/users/nova => generated 82 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:20:40.965617 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7d91666-e896-46f6-8028-ccae2ac00745 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:40.965617 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7d91666-e896-46f6-8028-ccae2ac00745 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:40.965877 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7d91666-e896-46f6-8028-ccae2ac00745 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:40.967238 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 249/496] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:20:40 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:20:40.967766 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7d91666-e896-46f6-8028-ccae2ac00745 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:40.994570 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-172d2c7f-dde5-4533-9469-f715d45f0c17 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:41.028940 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aff7e16-338a-46a6-9c29-97223b7cc06e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:41.029547 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aff7e16-338a-46a6-9c29-97223b7cc06e None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:41.029855 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aff7e16-338a-46a6-9c29-97223b7cc06e None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:41.030131 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aff7e16-338a-46a6-9c29-97223b7cc06e None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:41.030784 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aff7e16-338a-46a6-9c29-97223b7cc06e None admin] RBAC: Adding query filter params (name=service) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:41.031085 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aff7e16-338a-46a6-9c29-97223b7cc06e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:41.032434 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aff7e16-338a-46a6-9c29-97223b7cc06e None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:41.040109 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 248/497] 10.222.0.90 () {56 vars in 1103 bytes} [Mon Aug 29 21:20:40 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:41.041186 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71f540f9-a86f-4c8e-bbbb-97b428c2bd75 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:41.041468 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71f540f9-a86f-4c8e-bbbb-97b428c2bd75 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:41.041742 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71f540f9-a86f-4c8e-bbbb-97b428c2bd75 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:41.058845 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-71f540f9-a86f-4c8e-bbbb-97b428c2bd75 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:20:41.060020 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-71f540f9-a86f-4c8e-bbbb-97b428c2bd75 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:20:41.077551 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7d91666-e896-46f6-8028-ccae2ac00745 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:41.105908 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:41.111604 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f1f1df7-2252-4210-8ccf-13b0571e38bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:41.112246 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f1f1df7-2252-4210-8ccf-13b0571e38bc None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:41.112561 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f1f1df7-2252-4210-8ccf-13b0571e38bc None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:41.112837 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f1f1df7-2252-4210-8ccf-13b0571e38bc None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:41.123342 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f1f1df7-2252-4210-8ccf-13b0571e38bc None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:41.124984 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f1f1df7-2252-4210-8ccf-13b0571e38bc None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:41.127513 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 249/498] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:20:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 21:20:41.128868 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71f540f9-a86f-4c8e-bbbb-97b428c2bd75 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:41.142149 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-2f1f1df7-2252-4210-8ccf-13b0571e38bc None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:20:41.142149 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:41.142149 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:41.142149 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:41.142149 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:41.142149 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:41.142149 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:41.142149 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:41.142149 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:41.142149 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:41.142149 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:41.142149 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:41.142149 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 21:20:41.142149 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 21:20:41.142149 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 21:20:41.142149 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 21:20:41.143327 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:41.143327 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:41.143327 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:41.143327 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:41.143327 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:41.143327 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:41.143327 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:41.143327 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:41.143327 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:41.143327 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:41.143327 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:41.143327 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:41.143327 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:41.143327 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:41.143327 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:41.143327 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:41.143327 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 21:20:41.145172 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 21:20:41.145172 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:20:41.145172 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:20:41.145172 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 250/499] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:20:40 2022] GET /identity/v3/domains/Default => generated 87 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:20:41.145172 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9479c38-e258-4fb9-a0dd-50698bcef4df None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:41.145510 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9479c38-e258-4fb9-a0dd-50698bcef4df None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:41.145588 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9479c38-e258-4fb9-a0dd-50698bcef4df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:41.163725 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-c9479c38-e258-4fb9-a0dd-50698bcef4df None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:20:41.164925 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-c9479c38-e258-4fb9-a0dd-50698bcef4df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:20:41.212909 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:41.239527 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 251/500] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:20:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 21:20:41.240432 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9479c38-e258-4fb9-a0dd-50698bcef4df None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:41.248152 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71f540f9-a86f-4c8e-bbbb-97b428c2bd75 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:41.287970 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf67068f-8166-48fb-af26-e4bc5c172900 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:41.288666 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf67068f-8166-48fb-af26-e4bc5c172900 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:41.288953 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf67068f-8166-48fb-af26-e4bc5c172900 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:41.289227 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf67068f-8166-48fb-af26-e4bc5c172900 None admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:41.290256 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf67068f-8166-48fb-af26-e4bc5c172900 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:41.290565 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf67068f-8166-48fb-af26-e4bc5c172900 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:41.292879 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf67068f-8166-48fb-af26-e4bc5c172900 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:41.341082 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 250/501] 10.222.0.90 () {56 vars in 1133 bytes} [Mon Aug 29 21:20:41 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:41.343305 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf67068f-8166-48fb-af26-e4bc5c172900 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:41.369417 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9479c38-e258-4fb9-a0dd-50698bcef4df None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:41.386541 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:41.407525 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b44d93de-e7f2-427d-a7c8-11f9d5eea980 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:41.408481 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b44d93de-e7f2-427d-a7c8-11f9d5eea980 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:41.408988 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b44d93de-e7f2-427d-a7c8-11f9d5eea980 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:41.409475 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b44d93de-e7f2-427d-a7c8-11f9d5eea980 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:41.420048 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b44d93de-e7f2-427d-a7c8-11f9d5eea980 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:41.421497 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b44d93de-e7f2-427d-a7c8-11f9d5eea980 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:41.437719 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-b44d93de-e7f2-427d-a7c8-11f9d5eea980 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:20:41.437719 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:41.437719 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:41.437719 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:41.437719 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:41.437719 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:41.437719 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:41.437719 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:41.437719 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:41.437719 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:41.437719 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:41.437719 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:41.437719 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 21:20:41.437719 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 21:20:41.437719 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 21:20:41.437719 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 21:20:41.438695 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:41.438695 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:41.438695 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:41.438695 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:41.438695 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:41.438695 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:41.438695 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:41.438695 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:41.438695 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:41.438695 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:41.438695 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:41.438695 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:41.438695 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:41.438695 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:41.438695 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:41.438695 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:41.438695 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 21:20:41.439564 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 21:20:41.439564 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:20:41.439564 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:20:41.439564 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 252/502] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:20:41 2022] GET /identity/v3/domains/Default => generated 87 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:20:41.440612 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b44d93de-e7f2-427d-a7c8-11f9d5eea980 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:41.459808 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf67068f-8166-48fb-af26-e4bc5c172900 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:41.472389 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:41.496208 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69024ddf-89fd-430f-b287-4f82cc3d8410 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:41.496827 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69024ddf-89fd-430f-b287-4f82cc3d8410 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:41.497114 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69024ddf-89fd-430f-b287-4f82cc3d8410 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:41.497386 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69024ddf-89fd-430f-b287-4f82cc3d8410 None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:41.498054 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69024ddf-89fd-430f-b287-4f82cc3d8410 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:41.498355 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69024ddf-89fd-430f-b287-4f82cc3d8410 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:41.499735 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69024ddf-89fd-430f-b287-4f82cc3d8410 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:41.512670 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 251/503] 10.222.0.90 () {56 vars in 1109 bytes} [Mon Aug 29 21:20:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:41.513481 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69024ddf-89fd-430f-b287-4f82cc3d8410 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:41.552012 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b44d93de-e7f2-427d-a7c8-11f9d5eea980 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:41.585469 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0cee99a-5ef1-4924-a97f-b4f035c3a8e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:41.586110 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0cee99a-5ef1-4924-a97f-b4f035c3a8e0 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:41.586399 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0cee99a-5ef1-4924-a97f-b4f035c3a8e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:41.586672 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0cee99a-5ef1-4924-a97f-b4f035c3a8e0 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:41.612420 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:41.616875 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69024ddf-89fd-430f-b287-4f82cc3d8410 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:41.650759 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ece4478-ad9c-4131-886c-766885dbb0df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:41.651405 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ece4478-ad9c-4131-886c-766885dbb0df None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:41.651686 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ece4478-ad9c-4131-886c-766885dbb0df None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:41.651958 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ece4478-ad9c-4131-886c-766885dbb0df None admin] PATH_INFO: `/v3/roles/service` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:41.655711 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0cee99a-5ef1-4924-a97f-b4f035c3a8e0 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:41.657192 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0cee99a-5ef1-4924-a97f-b4f035c3a8e0 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:41.663690 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ece4478-ad9c-4131-886c-766885dbb0df None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 21:20:41.670034 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-2ece4478-ad9c-4131-886c-766885dbb0df None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 29 21:20:41.670034 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:41.670034 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:41.670034 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:41.670034 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:41.670034 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:41.670034 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:41.670034 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:41.670034 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:41.670034 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:41.670034 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:41.670034 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:41.670034 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 21:20:41.670034 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 21:20:41.670034 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 21:20:41.670034 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 21:20:41.670943 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 21:20:41.670943 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 21:20:41.670943 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 21:20:41.670943 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 21:20:41.670943 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:41.670943 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:41.670943 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:41.670943 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:41.670943 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:41.670943 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:41.670943 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:41.670943 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:41.670943 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:41.670943 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:41.670943 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:41.670943 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:41.671773 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:41.671773 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:41.671773 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:41.671773 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:41.671773 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 21:20:41.671773 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 21:20:41.671773 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 21:20:41.671773 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 21:20:41.671773 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 21:20:41.671773 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 21:20:41.671773 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 29 21:20:41.671773 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:20:41.671773 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 252/504] 10.222.0.90 () {56 vars in 1102 bytes} [Mon Aug 29 21:20:41 2022] GET /identity/v3/roles/service => generated 85 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:20:41.672475 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ece4478-ad9c-4131-886c-766885dbb0df None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:41.689920 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:41.761467 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 253/505] 10.222.0.90 () {58 vars in 1282 bytes} [Mon Aug 29 21:20:41 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 322 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 29 21:20:41.762354 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0cee99a-5ef1-4924-a97f-b4f035c3a8e0 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:41.779635 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ece4478-ad9c-4131-886c-766885dbb0df None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:41.812588 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0cf2329-d03a-40f8-a415-67cd95705770 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:41.813287 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0cf2329-d03a-40f8-a415-67cd95705770 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:41.813663 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0cf2329-d03a-40f8-a415-67cd95705770 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:41.814037 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0cf2329-d03a-40f8-a415-67cd95705770 None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:41.814686 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0cf2329-d03a-40f8-a415-67cd95705770 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:41.814998 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0cf2329-d03a-40f8-a415-67cd95705770 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:41.817388 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0cf2329-d03a-40f8-a415-67cd95705770 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:41.828889 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 253/506] 10.222.0.90 () {56 vars in 1109 bytes} [Mon Aug 29 21:20:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:41.829676 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0cf2329-d03a-40f8-a415-67cd95705770 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:41.849515 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:41.869930 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0cee99a-5ef1-4924-a97f-b4f035c3a8e0 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:41.898974 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:41.905199 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1e5afe7-ee79-456e-a050-569b012ac116 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:41.905774 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1e5afe7-ee79-456e-a050-569b012ac116 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:41.906081 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1e5afe7-ee79-456e-a050-569b012ac116 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:41.906354 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1e5afe7-ee79-456e-a050-569b012ac116 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:41.916506 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1e5afe7-ee79-456e-a050-569b012ac116 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:41.917985 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1e5afe7-ee79-456e-a050-569b012ac116 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:41.933043 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-d1e5afe7-ee79-456e-a050-569b012ac116 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:20:41.933043 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:41.933043 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:41.933043 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:41.933043 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:41.933043 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:41.933043 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:41.933043 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:41.933043 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:41.933043 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:41.933043 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:41.933043 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:41.933043 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 21:20:41.933043 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 21:20:41.933043 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 21:20:41.933043 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 21:20:41.934063 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:41.934063 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:41.934063 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:41.934063 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:41.934063 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:41.934063 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:41.934063 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:41.934063 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:41.934063 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:41.934063 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:41.934063 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:41.934063 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:41.934063 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:41.934063 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:41.934063 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:41.934063 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:41.934063 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 21:20:41.934960 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 21:20:41.934960 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:20:41.934960 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:20:41.935125 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0cf2329-d03a-40f8-a415-67cd95705770 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:41.935508 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 254/507] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:20:41 2022] GET /identity/v3/domains/Default => generated 87 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:20:41.935508 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1e5afe7-ee79-456e-a050-569b012ac116 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:41.969418 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cd98e8e-fa0c-4812-a405-7a6928d95415 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:41.970085 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cd98e8e-fa0c-4812-a405-7a6928d95415 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:41.970367 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cd98e8e-fa0c-4812-a405-7a6928d95415 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:41.970639 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cd98e8e-fa0c-4812-a405-7a6928d95415 None admin] PATH_INFO: `/v3/users/glance` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:41.977848 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cd98e8e-fa0c-4812-a405-7a6928d95415 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:41.979287 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cd98e8e-fa0c-4812-a405-7a6928d95415 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:41.992638 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-8cd98e8e-fa0c-4812-a405-7a6928d95415 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Aug 29 21:20:41.992638 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:41.992638 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:41.992638 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:41.992638 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:41.992638 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:41.992638 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:41.992638 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:41.992638 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:41.992638 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:41.992638 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:41.992638 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:41.992638 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 29 21:20:41.992638 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 29 21:20:41.992638 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 29 21:20:41.992638 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 29 21:20:41.993629 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:41.993629 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:41.993629 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 29 21:20:41.993629 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 21:20:41.993629 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 29 21:20:41.993629 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 21:20:41.993629 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:41.993629 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:41.993629 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:41.993629 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:41.993629 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:41.993629 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:41.993629 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:41.993629 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:41.993629 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:41.993629 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:41.993629 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:41.993629 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:41.995180 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:41.995180 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:41.995180 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 29 21:20:41.995180 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 29 21:20:41.995180 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 29 21:20:41.995180 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 29 21:20:41.995180 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 29 21:20:41.995180 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 29 21:20:41.995180 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Aug 29 21:20:41.995180 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:20:41.995180 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 254/508] 10.222.0.90 () {56 vars in 1099 bytes} [Mon Aug 29 21:20:41 2022] GET /identity/v3/users/glance => generated 84 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:20:41.995180 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cd98e8e-fa0c-4812-a405-7a6928d95415 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:42.042500 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1e5afe7-ee79-456e-a050-569b012ac116 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:42.075816 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:42.076005 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4c10612-6235-4de5-b9ea-1a18f7ed20d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:42.076496 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4c10612-6235-4de5-b9ea-1a18f7ed20d3 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:42.076787 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4c10612-6235-4de5-b9ea-1a18f7ed20d3 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:42.077066 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4c10612-6235-4de5-b9ea-1a18f7ed20d3 None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:42.077732 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4c10612-6235-4de5-b9ea-1a18f7ed20d3 None admin] RBAC: Adding query filter params (name=service) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:42.078071 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4c10612-6235-4de5-b9ea-1a18f7ed20d3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:42.079451 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4c10612-6235-4de5-b9ea-1a18f7ed20d3 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:42.087909 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 255/509] 10.222.0.90 () {56 vars in 1103 bytes} [Mon Aug 29 21:20:41 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:42.088705 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4c10612-6235-4de5-b9ea-1a18f7ed20d3 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:42.099355 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cd98e8e-fa0c-4812-a405-7a6928d95415 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:42.132387 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a96a7204-b10d-4328-88a9-1dbe4bc3bc19 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:42.133017 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a96a7204-b10d-4328-88a9-1dbe4bc3bc19 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:42.133293 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a96a7204-b10d-4328-88a9-1dbe4bc3bc19 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:42.133562 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a96a7204-b10d-4328-88a9-1dbe4bc3bc19 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:42.137488 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:42.144618 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a96a7204-b10d-4328-88a9-1dbe4bc3bc19 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:42.146165 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a96a7204-b10d-4328-88a9-1dbe4bc3bc19 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:42.160686 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-a96a7204-b10d-4328-88a9-1dbe4bc3bc19 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:20:42.160686 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:42.160686 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:42.160686 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:42.160686 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:42.160686 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:42.160686 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:42.160686 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:42.160686 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:42.160686 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:42.160686 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:42.160686 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:42.160686 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 21:20:42.160686 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 21:20:42.160686 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 21:20:42.160686 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 21:20:42.161721 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:42.161721 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:42.161721 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:42.161721 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:42.161721 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:42.161721 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:42.161721 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:42.161721 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:42.161721 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:42.161721 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:42.161721 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:42.161721 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:42.161721 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:42.161721 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:42.161721 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:42.161721 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:42.161721 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 21:20:42.162639 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 21:20:42.162639 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:20:42.162639 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:20:42.162639 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 255/510] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:20:41 2022] GET /identity/v3/domains/Default => generated 87 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:20:42.162639 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a96a7204-b10d-4328-88a9-1dbe4bc3bc19 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:42.193511 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4c10612-6235-4de5-b9ea-1a18f7ed20d3 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:42.227822 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd819cd2-966f-46c8-9282-78f2e773e059 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:42.228452 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd819cd2-966f-46c8-9282-78f2e773e059 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:42.228731 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd819cd2-966f-46c8-9282-78f2e773e059 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:42.229007 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd819cd2-966f-46c8-9282-78f2e773e059 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:42.236361 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd819cd2-966f-46c8-9282-78f2e773e059 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:42.238669 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd819cd2-966f-46c8-9282-78f2e773e059 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:42.249624 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-dd819cd2-966f-46c8-9282-78f2e773e059 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Aug 29 21:20:42.249624 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:42.249624 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:42.249624 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:42.249624 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:42.249624 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:42.249624 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:42.249624 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:42.249624 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:42.249624 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:42.249624 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:42.249624 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:42.249624 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 29 21:20:42.249624 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 29 21:20:42.249624 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 29 21:20:42.249624 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 29 21:20:42.250687 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:42.250687 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:42.250687 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 29 21:20:42.250687 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 21:20:42.250687 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 29 21:20:42.250687 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 21:20:42.250687 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:42.250687 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:42.250687 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:42.250687 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:42.250687 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:42.250687 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:42.250687 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:42.250687 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:42.250687 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:42.250687 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:42.250687 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:42.250687 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:42.251602 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:42.251602 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:42.251602 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 29 21:20:42.251602 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 29 21:20:42.251602 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 29 21:20:42.251602 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 29 21:20:42.251602 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 29 21:20:42.251602 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 29 21:20:42.251602 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Aug 29 21:20:42.251602 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:20:42.251602 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 256/511] 10.222.0.90 () {56 vars in 1099 bytes} [Mon Aug 29 21:20:42 2022] GET /identity/v3/users/cinder => generated 84 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:20:42.251602 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd819cd2-966f-46c8-9282-78f2e773e059 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:42.271635 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a96a7204-b10d-4328-88a9-1dbe4bc3bc19 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:42.300333 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:42.306641 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78424008-20f1-41f4-bb61-eaf5db783fb2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:42.307249 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78424008-20f1-41f4-bb61-eaf5db783fb2 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:42.307525 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78424008-20f1-41f4-bb61-eaf5db783fb2 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:42.307795 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78424008-20f1-41f4-bb61-eaf5db783fb2 None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:42.308429 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78424008-20f1-41f4-bb61-eaf5db783fb2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:42.308727 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78424008-20f1-41f4-bb61-eaf5db783fb2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:42.310123 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78424008-20f1-41f4-bb61-eaf5db783fb2 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:42.322329 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 256/512] 10.222.0.90 () {56 vars in 1109 bytes} [Mon Aug 29 21:20:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:42.323113 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78424008-20f1-41f4-bb61-eaf5db783fb2 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:42.354647 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd819cd2-966f-46c8-9282-78f2e773e059 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:42.356949 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:42.389438 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1078dd1-b612-48f1-8561-88b76dcde5db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:42.390078 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1078dd1-b612-48f1-8561-88b76dcde5db None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:42.390360 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1078dd1-b612-48f1-8561-88b76dcde5db None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:42.390639 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1078dd1-b612-48f1-8561-88b76dcde5db None admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:42.391367 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1078dd1-b612-48f1-8561-88b76dcde5db None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:42.391675 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1078dd1-b612-48f1-8561-88b76dcde5db None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:42.393113 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1078dd1-b612-48f1-8561-88b76dcde5db None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:42.428208 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78424008-20f1-41f4-bb61-eaf5db783fb2 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:42.429942 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 257/513] 10.222.0.90 () {56 vars in 1137 bytes} [Mon Aug 29 21:20:42 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:42.430866 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1078dd1-b612-48f1-8561-88b76dcde5db None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:42.465316 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaf0f7b7-98e5-4583-9332-074e852a79b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:42.465988 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaf0f7b7-98e5-4583-9332-074e852a79b0 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:42.466262 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaf0f7b7-98e5-4583-9332-074e852a79b0 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:42.466538 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaf0f7b7-98e5-4583-9332-074e852a79b0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:42.477307 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaf0f7b7-98e5-4583-9332-074e852a79b0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:42.478817 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaf0f7b7-98e5-4583-9332-074e852a79b0 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:42.493458 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-aaf0f7b7-98e5-4583-9332-074e852a79b0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:20:42.493458 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:42.493458 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:42.493458 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:42.493458 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:42.493458 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:42.493458 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:42.493458 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:42.493458 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:42.493458 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:42.493458 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:42.493458 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:42.493458 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 21:20:42.493458 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 21:20:42.493458 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 21:20:42.493458 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 21:20:42.494412 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:42.494412 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:42.494412 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:42.494412 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:42.494412 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:42.494412 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:42.494412 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:42.494412 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:42.494412 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:42.494412 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:42.494412 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:42.494412 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:42.494412 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:42.494412 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:42.494412 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:42.494412 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:42.494412 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 21:20:42.495286 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 21:20:42.495286 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:20:42.495286 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:20:42.495286 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 257/514] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:20:42 2022] GET /identity/v3/domains/Default => generated 87 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:20:42.495512 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaf0f7b7-98e5-4583-9332-074e852a79b0 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:42.534818 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1078dd1-b612-48f1-8561-88b76dcde5db None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:42.539164 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:42.569894 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e711d649-cca3-44f5-96bd-1a9bbcd0e1c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:42.570504 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e711d649-cca3-44f5-96bd-1a9bbcd0e1c9 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:42.570790 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e711d649-cca3-44f5-96bd-1a9bbcd0e1c9 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:42.571069 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e711d649-cca3-44f5-96bd-1a9bbcd0e1c9 None admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:42.571716 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e711d649-cca3-44f5-96bd-1a9bbcd0e1c9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:42.572018 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e711d649-cca3-44f5-96bd-1a9bbcd0e1c9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:42.573390 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e711d649-cca3-44f5-96bd-1a9bbcd0e1c9 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:42.585049 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 258/515] 10.222.0.90 () {56 vars in 1109 bytes} [Mon Aug 29 21:20:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:42.585865 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e711d649-cca3-44f5-96bd-1a9bbcd0e1c9 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:42.595950 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaf0f7b7-98e5-4583-9332-074e852a79b0 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:42.611761 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:42.631997 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88553ee9-a986-43ac-83a7-878be58eb71e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:42.632615 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88553ee9-a986-43ac-83a7-878be58eb71e None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:42.632898 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88553ee9-a986-43ac-83a7-878be58eb71e None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:42.633175 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88553ee9-a986-43ac-83a7-878be58eb71e None admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:42.633951 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88553ee9-a986-43ac-83a7-878be58eb71e None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:42.634268 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88553ee9-a986-43ac-83a7-878be58eb71e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:42.635708 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88553ee9-a986-43ac-83a7-878be58eb71e None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:42.657831 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 258/516] 10.222.0.90 () {56 vars in 1137 bytes} [Mon Aug 29 21:20:42 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:42.657831 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88553ee9-a986-43ac-83a7-878be58eb71e None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:42.688783 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e711d649-cca3-44f5-96bd-1a9bbcd0e1c9 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:42.723397 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1970a44-9aef-4d4e-854f-50c46d5a2509 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:42.724000 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1970a44-9aef-4d4e-854f-50c46d5a2509 None admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:42.724281 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1970a44-9aef-4d4e-854f-50c46d5a2509 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:42.724554 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1970a44-9aef-4d4e-854f-50c46d5a2509 None admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:42.725200 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1970a44-9aef-4d4e-854f-50c46d5a2509 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:42.726810 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1970a44-9aef-4d4e-854f-50c46d5a2509 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:42.747632 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 259/517] 10.222.0.90 () {60 vars in 1132 bytes} [Mon Aug 29 21:20:42 2022] POST /identity/v3/users => generated 247 bytes in 162 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:20:42.748433 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1970a44-9aef-4d4e-854f-50c46d5a2509 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:42.756817 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88553ee9-a986-43ac-83a7-878be58eb71e None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:42.758769 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:42.798948 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74268cf5-19d5-49fe-a7b4-2ee1d296a7f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:42.799585 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74268cf5-19d5-49fe-a7b4-2ee1d296a7f9 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:42.800446 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74268cf5-19d5-49fe-a7b4-2ee1d296a7f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:42.800697 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74268cf5-19d5-49fe-a7b4-2ee1d296a7f9 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:42.849458 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:42.880700 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74268cf5-19d5-49fe-a7b4-2ee1d296a7f9 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:42.882240 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74268cf5-19d5-49fe-a7b4-2ee1d296a7f9 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:42.882737 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1970a44-9aef-4d4e-854f-50c46d5a2509 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:42.920520 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f82fcb80-cf62-45a3-a163-8168d885083d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:42.921224 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f82fcb80-cf62-45a3-a163-8168d885083d None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:42.921582 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f82fcb80-cf62-45a3-a163-8168d885083d None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:42.921897 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f82fcb80-cf62-45a3-a163-8168d885083d None admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:42.922968 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f82fcb80-cf62-45a3-a163-8168d885083d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:42.922968 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f82fcb80-cf62-45a3-a163-8168d885083d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:42.924249 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f82fcb80-cf62-45a3-a163-8168d885083d None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:42.941905 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 260/518] 10.222.0.90 () {56 vars in 1109 bytes} [Mon Aug 29 21:20:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:42.941905 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f82fcb80-cf62-45a3-a163-8168d885083d None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:43.000236 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:43.005338 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 259/519] 10.222.0.90 () {58 vars in 1282 bytes} [Mon Aug 29 21:20:42 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 351 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 29 21:20:43.006670 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74268cf5-19d5-49fe-a7b4-2ee1d296a7f9 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:43.056642 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f82fcb80-cf62-45a3-a163-8168d885083d None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:43.094121 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebdb195a-18db-43e5-9b5c-012446b987fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:43.094762 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebdb195a-18db-43e5-9b5c-012446b987fd None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:43.095044 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebdb195a-18db-43e5-9b5c-012446b987fd None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:43.095349 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebdb195a-18db-43e5-9b5c-012446b987fd None admin] PATH_INFO: `/v3/projects/service` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:43.103500 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebdb195a-18db-43e5-9b5c-012446b987fd None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:43.104754 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebdb195a-18db-43e5-9b5c-012446b987fd None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:43.106663 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:43.116945 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74268cf5-19d5-49fe-a7b4-2ee1d296a7f9 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:43.120639 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-ebdb195a-18db-43e5-9b5c-012446b987fd None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 21:20:43.120639 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:43.120639 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:43.120639 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:43.120639 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:43.120639 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:43.120639 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:43.120639 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:43.120639 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:43.120639 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:43.120639 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:43.120639 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:43.120639 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 29 21:20:43.120639 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 29 21:20:43.120639 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 29 21:20:43.121511 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 29 21:20:43.121511 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:43.121511 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:43.121511 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:43.121511 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:43.121511 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:43.121511 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:43.121511 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:43.121511 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:43.121511 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:43.121511 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:43.121511 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:43.121511 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:43.121511 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:43.121511 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:43.121511 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:43.121511 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:43.127872 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 29 21:20:43.127872 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 29 21:20:43.127872 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 29 21:20:43.127872 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 29 21:20:43.127872 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 29 21:20:43.127872 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 29 21:20:43.127872 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 21:20:43.127872 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:20:43.127872 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 261/520] 10.222.0.90 () {56 vars in 1111 bytes} [Mon Aug 29 21:20:42 2022] GET /identity/v3/projects/service => generated 88 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:20:43.127872 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebdb195a-18db-43e5-9b5c-012446b987fd None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:43.157375 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0f4c97f-5282-4d76-8396-008b06634f17 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:43.158566 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0f4c97f-5282-4d76-8396-008b06634f17 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:43.159154 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0f4c97f-5282-4d76-8396-008b06634f17 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:43.160258 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0f4c97f-5282-4d76-8396-008b06634f17 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:43.233285 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:43.233599 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebdb195a-18db-43e5-9b5c-012446b987fd None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:43.236077 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0f4c97f-5282-4d76-8396-008b06634f17 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:43.237767 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0f4c97f-5282-4d76-8396-008b06634f17 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:43.272036 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75aa4d01-4861-4375-806c-222cd0639e92 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:43.272821 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75aa4d01-4861-4375-806c-222cd0639e92 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:43.273048 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75aa4d01-4861-4375-806c-222cd0639e92 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:43.273319 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75aa4d01-4861-4375-806c-222cd0639e92 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:43.281096 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75aa4d01-4861-4375-806c-222cd0639e92 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:43.282595 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75aa4d01-4861-4375-806c-222cd0639e92 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:43.294578 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-75aa4d01-4861-4375-806c-222cd0639e92 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Aug 29 21:20:43.294578 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:43.294578 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:43.294578 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:43.294578 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:43.294578 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:43.294578 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:43.294578 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:43.294578 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:43.294578 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:43.294578 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:43.294578 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:43.294578 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 29 21:20:43.294578 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 29 21:20:43.294578 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 29 21:20:43.294578 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 29 21:20:43.295554 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:43.295554 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:43.295554 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 29 21:20:43.295554 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 21:20:43.295554 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 29 21:20:43.295554 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 21:20:43.295554 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:43.295554 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:43.295554 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:43.295554 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:43.295554 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:43.295554 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:43.295554 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:43.295554 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:43.295554 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:43.295554 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:43.295554 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:43.295554 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:43.296484 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:43.296484 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:43.296484 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 29 21:20:43.296484 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 29 21:20:43.296484 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 29 21:20:43.296484 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 29 21:20:43.296484 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 29 21:20:43.296484 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 29 21:20:43.296484 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Aug 29 21:20:43.296484 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:20:43.296484 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 262/521] 10.222.0.90 () {56 vars in 1102 bytes} [Mon Aug 29 21:20:43 2022] GET /identity/v3/users/neutron => generated 85 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:20:43.297079 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75aa4d01-4861-4375-806c-222cd0639e92 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:43.352975 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 260/522] 10.222.0.90 () {58 vars in 1282 bytes} [Mon Aug 29 21:20:43 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 347 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 29 21:20:43.353943 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0f4c97f-5282-4d76-8396-008b06634f17 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:43.365703 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:43.411024 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75aa4d01-4861-4375-806c-222cd0639e92 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:43.449389 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1f7561e-1363-4610-8334-01df5e4beeab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:43.450053 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1f7561e-1363-4610-8334-01df5e4beeab None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:43.450338 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1f7561e-1363-4610-8334-01df5e4beeab None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:43.450606 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1f7561e-1363-4610-8334-01df5e4beeab None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:43.462094 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1f7561e-1363-4610-8334-01df5e4beeab None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:43.463527 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1f7561e-1363-4610-8334-01df5e4beeab None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:43.468657 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:43.471956 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0f4c97f-5282-4d76-8396-008b06634f17 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:43.479947 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-a1f7561e-1363-4610-8334-01df5e4beeab None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:20:43.479947 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:43.479947 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:43.479947 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:43.479947 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:43.479947 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:43.479947 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:43.479947 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:43.479947 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:43.479947 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:43.479947 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:43.479947 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:43.479947 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 21:20:43.479947 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 21:20:43.479947 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 21:20:43.479947 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 21:20:43.480847 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:43.480847 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:43.480847 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:43.480847 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:43.480847 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:43.480847 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:43.480847 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:43.480847 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:43.480847 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:43.480847 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:43.480847 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:43.480847 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:43.480847 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:43.480847 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:43.480847 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:43.480847 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:43.480847 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 21:20:43.481916 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 21:20:43.481916 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:20:43.481916 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:20:43.481916 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 263/523] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:20:43 2022] GET /identity/v3/domains/Default => generated 87 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:20:43.481916 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1f7561e-1363-4610-8334-01df5e4beeab None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:43.517241 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81f115cc-4eac-4eb0-a217-7de2583bc517 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:43.517902 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81f115cc-4eac-4eb0-a217-7de2583bc517 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:43.518183 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81f115cc-4eac-4eb0-a217-7de2583bc517 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:43.518568 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81f115cc-4eac-4eb0-a217-7de2583bc517 None admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:43.519487 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81f115cc-4eac-4eb0-a217-7de2583bc517 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:43.519794 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81f115cc-4eac-4eb0-a217-7de2583bc517 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:43.521270 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81f115cc-4eac-4eb0-a217-7de2583bc517 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:43.544475 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 261/524] 10.222.0.90 () {56 vars in 1148 bytes} [Mon Aug 29 21:20:43 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:43.544475 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81f115cc-4eac-4eb0-a217-7de2583bc517 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:43.590618 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1f7561e-1363-4610-8334-01df5e4beeab None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:43.601626 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:43.627515 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-236279cb-f6a6-43b7-b601-550af23a2a06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:43.628227 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-236279cb-f6a6-43b7-b601-550af23a2a06 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:43.628733 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-236279cb-f6a6-43b7-b601-550af23a2a06 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:43.629016 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-236279cb-f6a6-43b7-b601-550af23a2a06 None admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:43.629749 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-236279cb-f6a6-43b7-b601-550af23a2a06 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:43.630113 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-236279cb-f6a6-43b7-b601-550af23a2a06 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:43.631557 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-236279cb-f6a6-43b7-b601-550af23a2a06 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:43.649089 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81f115cc-4eac-4eb0-a217-7de2583bc517 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:43.654510 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 264/525] 10.222.0.90 () {56 vars in 1139 bytes} [Mon Aug 29 21:20:43 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:43.654510 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-236279cb-f6a6-43b7-b601-550af23a2a06 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:43.689483 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0188b75-15a3-40b9-baa5-d62f251db9b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:43.690142 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0188b75-15a3-40b9-baa5-d62f251db9b2 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:43.690420 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0188b75-15a3-40b9-baa5-d62f251db9b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:43.690831 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0188b75-15a3-40b9-baa5-d62f251db9b2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:43.701749 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0188b75-15a3-40b9-baa5-d62f251db9b2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:43.703376 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0188b75-15a3-40b9-baa5-d62f251db9b2 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:43.720483 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-a0188b75-15a3-40b9-baa5-d62f251db9b2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:20:43.720483 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:43.720483 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:43.720483 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:43.720483 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:43.720483 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:43.720483 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:43.720483 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:43.720483 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:43.720483 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:43.720483 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:43.720483 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:43.720483 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 21:20:43.720483 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 21:20:43.720483 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 21:20:43.720483 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 21:20:43.722095 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:43.722095 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:43.722095 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:43.722095 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:43.722095 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:43.722095 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:43.722095 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:43.722095 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:43.722095 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:43.722095 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:43.722095 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:43.722095 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:43.722095 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:43.722095 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:43.722095 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:43.722095 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:43.722095 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 21:20:43.723000 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 21:20:43.723000 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:20:43.723000 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:20:43.723000 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 262/526] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:20:43 2022] GET /identity/v3/domains/Default => generated 87 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:20:43.723235 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0188b75-15a3-40b9-baa5-d62f251db9b2 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:43.740567 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:43.765823 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-236279cb-f6a6-43b7-b601-550af23a2a06 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:43.803602 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f0e1976-0952-466a-91ac-205b22beae66 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:43.804223 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f0e1976-0952-466a-91ac-205b22beae66 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:43.804503 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f0e1976-0952-466a-91ac-205b22beae66 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:43.804776 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f0e1976-0952-466a-91ac-205b22beae66 None admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:43.805421 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f0e1976-0952-466a-91ac-205b22beae66 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:43.805723 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f0e1976-0952-466a-91ac-205b22beae66 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:43.807138 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f0e1976-0952-466a-91ac-205b22beae66 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:43.819496 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 265/527] 10.222.0.90 () {56 vars in 1109 bytes} [Mon Aug 29 21:20:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:43.820312 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f0e1976-0952-466a-91ac-205b22beae66 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:43.837247 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0188b75-15a3-40b9-baa5-d62f251db9b2 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:43.853115 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:43.879064 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d1f8a8e-ee5f-44f6-b318-db9136ad2e79 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:43.879494 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1f8a8e-ee5f-44f6-b318-db9136ad2e79 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:43.879832 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1f8a8e-ee5f-44f6-b318-db9136ad2e79 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:43.880114 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1f8a8e-ee5f-44f6-b318-db9136ad2e79 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:43.880867 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d1f8a8e-ee5f-44f6-b318-db9136ad2e79 None admin] RBAC: Adding query filter params (role.id=a94bdb4c5796428baf63644c062dfcb2, scope.project.id=ec03e112e3d547fbab6faf7b485c3fca, user.id=a35b5b7dea7b421da056f063d2436ffb) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:43.881178 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d1f8a8e-ee5f-44f6-b318-db9136ad2e79 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:43.882876 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d1f8a8e-ee5f-44f6-b318-db9136ad2e79 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:43.890037 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 263/528] 10.222.0.90 () {56 vars in 1374 bytes} [Mon Aug 29 21:20:43 2022] GET /identity/v3/role_assignments?user.id=a35b5b7dea7b421da056f063d2436ffb&scope.project.id=ec03e112e3d547fbab6faf7b485c3fca&role.id=a94bdb4c5796428baf63644c062dfcb2 => generated 259 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:43.891000 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d1f8a8e-ee5f-44f6-b318-db9136ad2e79 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:43.953562 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f0e1976-0952-466a-91ac-205b22beae66 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:43.996905 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ac75e9e-9cdb-408b-8a0c-eff7aa4d7f16 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:43.998044 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ac75e9e-9cdb-408b-8a0c-eff7aa4d7f16 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:43.998530 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ac75e9e-9cdb-408b-8a0c-eff7aa4d7f16 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:43.999177 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ac75e9e-9cdb-408b-8a0c-eff7aa4d7f16 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:44.014092 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:44.048354 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d1f8a8e-ee5f-44f6-b318-db9136ad2e79 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:44.086964 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ac75e9e-9cdb-408b-8a0c-eff7aa4d7f16 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:44.088754 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ac75e9e-9cdb-408b-8a0c-eff7aa4d7f16 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:44.092623 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebb77774-259a-455c-8b94-304bc1193519 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:44.093256 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb77774-259a-455c-8b94-304bc1193519 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:44.093543 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb77774-259a-455c-8b94-304bc1193519 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:44.093875 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb77774-259a-455c-8b94-304bc1193519 None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:44.094579 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebb77774-259a-455c-8b94-304bc1193519 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:44.094889 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebb77774-259a-455c-8b94-304bc1193519 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:44.096598 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebb77774-259a-455c-8b94-304bc1193519 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:44.117713 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 264/529] 10.222.0.90 () {56 vars in 1109 bytes} [Mon Aug 29 21:20:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 223 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:44.118654 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebb77774-259a-455c-8b94-304bc1193519 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:44.126148 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:44.214683 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 266/530] 10.222.0.90 () {58 vars in 1282 bytes} [Mon Aug 29 21:20:43 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 394 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 29 21:20:44.216496 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ac75e9e-9cdb-408b-8a0c-eff7aa4d7f16 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:44.231240 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebb77774-259a-455c-8b94-304bc1193519 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:44.278355 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-068eb5a3-bc9b-47f7-985e-6bc1918ce71d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:44.279889 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-068eb5a3-bc9b-47f7-985e-6bc1918ce71d None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:44.280389 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-068eb5a3-bc9b-47f7-985e-6bc1918ce71d None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:44.281121 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-068eb5a3-bc9b-47f7-985e-6bc1918ce71d None admin] PATH_INFO: `/v3/projects/service` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:44.285698 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:44.294525 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-068eb5a3-bc9b-47f7-985e-6bc1918ce71d None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:44.297135 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-068eb5a3-bc9b-47f7-985e-6bc1918ce71d None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:44.314289 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-068eb5a3-bc9b-47f7-985e-6bc1918ce71d None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 21:20:44.314289 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:44.314289 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:44.314289 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:44.314289 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:44.314289 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:44.314289 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:44.314289 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:44.314289 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:44.314289 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:44.314289 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:44.314289 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:44.314289 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 29 21:20:44.314289 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 29 21:20:44.314289 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 29 21:20:44.315405 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 29 21:20:44.315405 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:44.315405 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:44.315405 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:44.315405 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:44.315405 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:44.315405 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:44.315405 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:44.315405 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:44.315405 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:44.315405 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:44.315405 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:44.315405 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:44.315405 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:44.315405 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:44.315405 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:44.315405 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:44.316658 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 29 21:20:44.316658 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 29 21:20:44.316658 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 29 21:20:44.316658 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 29 21:20:44.316658 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 29 21:20:44.316658 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 29 21:20:44.316658 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 21:20:44.316658 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:20:44.319114 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 265/531] 10.222.0.90 () {56 vars in 1111 bytes} [Mon Aug 29 21:20:44 2022] GET /identity/v3/projects/service => generated 88 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:20:44.320080 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-068eb5a3-bc9b-47f7-985e-6bc1918ce71d None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:44.342832 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ac75e9e-9cdb-408b-8a0c-eff7aa4d7f16 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:44.381539 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:44.394760 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5376d795-7dc1-4ecb-93d3-032bdc2a5f7b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:44.395634 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5376d795-7dc1-4ecb-93d3-032bdc2a5f7b None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:44.396668 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5376d795-7dc1-4ecb-93d3-032bdc2a5f7b None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:44.397095 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5376d795-7dc1-4ecb-93d3-032bdc2a5f7b None admin] PATH_INFO: `/v3/projects/service` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:44.406402 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5376d795-7dc1-4ecb-93d3-032bdc2a5f7b None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:44.408526 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5376d795-7dc1-4ecb-93d3-032bdc2a5f7b None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:44.424708 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-5376d795-7dc1-4ecb-93d3-032bdc2a5f7b None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 21:20:44.424708 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:44.424708 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:44.424708 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:44.424708 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:44.424708 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:44.424708 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:44.424708 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:44.424708 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:44.424708 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:44.424708 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:44.424708 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:44.424708 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 29 21:20:44.424708 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 29 21:20:44.424708 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 29 21:20:44.426018 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 29 21:20:44.426018 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:44.426018 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:44.426018 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:44.426018 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:44.426018 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:44.426018 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:44.426018 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:44.426018 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:44.426018 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:44.426018 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:44.426018 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:44.426018 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:44.426018 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:44.426018 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:44.426018 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:44.426018 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:44.426910 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 29 21:20:44.426910 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 29 21:20:44.426910 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 29 21:20:44.426910 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 29 21:20:44.426910 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 29 21:20:44.426910 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 29 21:20:44.426910 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 21:20:44.426910 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:20:44.426910 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 267/532] 10.222.0.90 () {56 vars in 1111 bytes} [Mon Aug 29 21:20:44 2022] GET /identity/v3/projects/service => generated 88 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:20:44.427397 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5376d795-7dc1-4ecb-93d3-032bdc2a5f7b None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:44.444975 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-068eb5a3-bc9b-47f7-985e-6bc1918ce71d None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:44.484169 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db403473-81ba-4933-afd6-b725646b8619 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:44.484970 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db403473-81ba-4933-afd6-b725646b8619 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:44.485184 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db403473-81ba-4933-afd6-b725646b8619 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:44.485459 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db403473-81ba-4933-afd6-b725646b8619 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:44.498736 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db403473-81ba-4933-afd6-b725646b8619 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:44.500948 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db403473-81ba-4933-afd6-b725646b8619 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:44.522030 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-db403473-81ba-4933-afd6-b725646b8619 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:20:44.522030 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:44.522030 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:44.522030 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:44.522030 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:44.522030 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:44.522030 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:44.522030 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:44.522030 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:44.522030 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:44.522030 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:44.522030 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:44.522030 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 21:20:44.522030 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 21:20:44.522030 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 21:20:44.522030 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 21:20:44.523071 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:44.523071 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:44.523071 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:44.523071 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:44.523071 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:44.523071 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:44.523071 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:44.523071 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:44.523071 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:44.523071 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:44.523071 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:44.523071 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:44.523071 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:44.523071 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:44.523071 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:44.523071 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:44.523071 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 21:20:44.523935 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 21:20:44.523935 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:20:44.523935 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:20:44.526565 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 266/533] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:20:44 2022] GET /identity/v3/domains/Default => generated 87 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:20:44.527592 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db403473-81ba-4933-afd6-b725646b8619 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:44.556857 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5376d795-7dc1-4ecb-93d3-032bdc2a5f7b None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:44.599012 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-244dc205-611d-4568-b10e-d2c0aa04bb43 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:44.599644 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-244dc205-611d-4568-b10e-d2c0aa04bb43 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:44.599870 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-244dc205-611d-4568-b10e-d2c0aa04bb43 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:44.600105 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-244dc205-611d-4568-b10e-d2c0aa04bb43 None admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:44.600780 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:44.600888 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-244dc205-611d-4568-b10e-d2c0aa04bb43 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:44.601090 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-244dc205-611d-4568-b10e-d2c0aa04bb43 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:44.602545 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-244dc205-611d-4568-b10e-d2c0aa04bb43 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:44.620938 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 268/534] 10.222.0.90 () {56 vars in 1148 bytes} [Mon Aug 29 21:20:44 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:44.621923 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-244dc205-611d-4568-b10e-d2c0aa04bb43 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:44.671864 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db403473-81ba-4933-afd6-b725646b8619 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:44.694030 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:44.711221 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf449576-75e7-42e1-ab9d-b0f7be3dc99d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:44.711840 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf449576-75e7-42e1-ab9d-b0f7be3dc99d None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:44.712088 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf449576-75e7-42e1-ab9d-b0f7be3dc99d None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:44.712364 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf449576-75e7-42e1-ab9d-b0f7be3dc99d None admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:44.713057 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf449576-75e7-42e1-ab9d-b0f7be3dc99d None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:44.713361 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf449576-75e7-42e1-ab9d-b0f7be3dc99d None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:44.715012 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf449576-75e7-42e1-ab9d-b0f7be3dc99d None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:44.733628 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 267/535] 10.222.0.90 () {56 vars in 1148 bytes} [Mon Aug 29 21:20:44 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:44.734642 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf449576-75e7-42e1-ab9d-b0f7be3dc99d None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:44.766910 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-244dc205-611d-4568-b10e-d2c0aa04bb43 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:44.804703 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41a48879-18ff-46e8-a713-a97fbd90bf93 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:44.814889 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41a48879-18ff-46e8-a713-a97fbd90bf93 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:44.814889 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41a48879-18ff-46e8-a713-a97fbd90bf93 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:44.814889 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41a48879-18ff-46e8-a713-a97fbd90bf93 None admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:44.814889 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41a48879-18ff-46e8-a713-a97fbd90bf93 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:44.814889 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41a48879-18ff-46e8-a713-a97fbd90bf93 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:44.814889 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41a48879-18ff-46e8-a713-a97fbd90bf93 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:44.820811 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 269/536] 10.222.0.90 () {56 vars in 1109 bytes} [Mon Aug 29 21:20:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:44.821627 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c1149e7-6420-488e-b226-a9b20d887f55 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:44.822288 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c1149e7-6420-488e-b226-a9b20d887f55 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:44.823177 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c1149e7-6420-488e-b226-a9b20d887f55 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:44.824543 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 270/537] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:20:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:20:44.825202 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c1149e7-6420-488e-b226-a9b20d887f55 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:44.871772 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf449576-75e7-42e1-ab9d-b0f7be3dc99d None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:44.882656 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:44.910270 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93cbc9d1-f1db-451a-bfc8-3aaf48a19e9d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:44.911231 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93cbc9d1-f1db-451a-bfc8-3aaf48a19e9d None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:44.911723 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93cbc9d1-f1db-451a-bfc8-3aaf48a19e9d None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:44.912095 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93cbc9d1-f1db-451a-bfc8-3aaf48a19e9d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:44.912832 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93cbc9d1-f1db-451a-bfc8-3aaf48a19e9d None admin] RBAC: Adding query filter params (role.id=a94bdb4c5796428baf63644c062dfcb2, scope.project.id=ec03e112e3d547fbab6faf7b485c3fca, user.id=c63a4eaf48ad43529f0aa328f2ebfd29) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:44.913196 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93cbc9d1-f1db-451a-bfc8-3aaf48a19e9d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:44.914715 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93cbc9d1-f1db-451a-bfc8-3aaf48a19e9d None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:44.925878 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 268/538] 10.222.0.90 () {56 vars in 1374 bytes} [Mon Aug 29 21:20:44 2022] GET /identity/v3/role_assignments?user.id=c63a4eaf48ad43529f0aa328f2ebfd29&scope.project.id=ec03e112e3d547fbab6faf7b485c3fca&role.id=a94bdb4c5796428baf63644c062dfcb2 => generated 259 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:44.925878 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93cbc9d1-f1db-451a-bfc8-3aaf48a19e9d None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:44.959562 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c1149e7-6420-488e-b226-a9b20d887f55 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:44.974886 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:45.002975 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abbf8ab5-b6a9-4f4b-9730-192e7fcdbf35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:45.004378 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abbf8ab5-b6a9-4f4b-9730-192e7fcdbf35 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:45.004708 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abbf8ab5-b6a9-4f4b-9730-192e7fcdbf35 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:45.004986 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abbf8ab5-b6a9-4f4b-9730-192e7fcdbf35 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:45.005762 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abbf8ab5-b6a9-4f4b-9730-192e7fcdbf35 None admin] RBAC: Adding query filter params (role.id=a94bdb4c5796428baf63644c062dfcb2, scope.project.id=ec03e112e3d547fbab6faf7b485c3fca, user.id=0dc226f7ddcb49ecacb51ce57fdfc11b) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:45.006105 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abbf8ab5-b6a9-4f4b-9730-192e7fcdbf35 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:45.007585 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abbf8ab5-b6a9-4f4b-9730-192e7fcdbf35 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:45.014593 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 271/539] 10.222.0.90 () {56 vars in 1374 bytes} [Mon Aug 29 21:20:44 2022] GET /identity/v3/role_assignments?user.id=0dc226f7ddcb49ecacb51ce57fdfc11b&scope.project.id=ec03e112e3d547fbab6faf7b485c3fca&role.id=a94bdb4c5796428baf63644c062dfcb2 => generated 259 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:45.016008 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91685f30-ca39-4756-a49a-4574285630ee None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:45.016297 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91685f30-ca39-4756-a49a-4574285630ee None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:45.016573 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91685f30-ca39-4756-a49a-4574285630ee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:45.038184 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-91685f30-ca39-4756-a49a-4574285630ee None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:20:45.039988 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-91685f30-ca39-4756-a49a-4574285630ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:20:45.090414 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93cbc9d1-f1db-451a-bfc8-3aaf48a19e9d None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:45.130692 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 272/540] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:20:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 21:20:45.141878 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9fc9011-a6c3-4615-ad37-f1ee118cd7af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:45.142536 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9fc9011-a6c3-4615-ad37-f1ee118cd7af None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:45.142816 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9fc9011-a6c3-4615-ad37-f1ee118cd7af None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:45.143087 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9fc9011-a6c3-4615-ad37-f1ee118cd7af None admin] PATH_INFO: `/v3/projects/service` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:45.151812 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9fc9011-a6c3-4615-ad37-f1ee118cd7af None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:45.153297 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9fc9011-a6c3-4615-ad37-f1ee118cd7af None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:45.166714 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-f9fc9011-a6c3-4615-ad37-f1ee118cd7af None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 21:20:45.166714 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:45.166714 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:45.166714 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:45.166714 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:45.166714 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:45.166714 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:45.166714 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:45.166714 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:45.166714 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:45.166714 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:45.166714 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:45.166714 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 29 21:20:45.166714 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 29 21:20:45.166714 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 29 21:20:45.167706 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 29 21:20:45.167706 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:45.167706 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:45.167706 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:45.167706 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:45.167706 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:45.167706 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:45.167706 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:45.167706 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:45.167706 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:45.167706 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:45.167706 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:45.167706 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:45.167706 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:45.167706 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:45.167706 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:45.167706 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:45.169209 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 29 21:20:45.169209 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 29 21:20:45.169209 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 29 21:20:45.169209 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 29 21:20:45.169209 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 29 21:20:45.169209 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 29 21:20:45.169209 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 21:20:45.169209 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:20:45.169209 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 269/541] 10.222.0.90 () {56 vars in 1111 bytes} [Mon Aug 29 21:20:44 2022] GET /identity/v3/projects/service => generated 88 bytes in 247 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:20:45.176101 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91685f30-ca39-4756-a49a-4574285630ee None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:45.287970 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91685f30-ca39-4756-a49a-4574285630ee None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:45.289538 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:45.334917 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd155d2c-441f-41cc-b267-053c09af5b11 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:45.335594 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd155d2c-441f-41cc-b267-053c09af5b11 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:45.335872 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd155d2c-441f-41cc-b267-053c09af5b11 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:45.336144 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd155d2c-441f-41cc-b267-053c09af5b11 None admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:45.336935 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd155d2c-441f-41cc-b267-053c09af5b11 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:45.337238 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd155d2c-441f-41cc-b267-053c09af5b11 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:45.338692 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd155d2c-441f-41cc-b267-053c09af5b11 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:45.357516 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 273/542] 10.222.0.90 () {56 vars in 1148 bytes} [Mon Aug 29 21:20:45 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:45.364045 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9fc9011-a6c3-4615-ad37-f1ee118cd7af None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:45.415573 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:45.464692 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd155d2c-441f-41cc-b267-053c09af5b11 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:45.492061 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9fc9011-a6c3-4615-ad37-f1ee118cd7af None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:45.527840 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b419b528-a9fa-4b1e-9f0b-00b3c980f673 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:45.528568 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b419b528-a9fa-4b1e-9f0b-00b3c980f673 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:45.528848 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b419b528-a9fa-4b1e-9f0b-00b3c980f673 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:45.529115 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b419b528-a9fa-4b1e-9f0b-00b3c980f673 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:45.529856 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b419b528-a9fa-4b1e-9f0b-00b3c980f673 None admin] RBAC: Adding query filter params (role.id=a94bdb4c5796428baf63644c062dfcb2, scope.project.id=ec03e112e3d547fbab6faf7b485c3fca, user.id=0e69b7f0c3574abc9de5fb580d22152a) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:45.530171 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b419b528-a9fa-4b1e-9f0b-00b3c980f673 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:45.531716 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b419b528-a9fa-4b1e-9f0b-00b3c980f673 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:45.538320 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 270/543] 10.222.0.90 () {56 vars in 1374 bytes} [Mon Aug 29 21:20:45 2022] GET /identity/v3/role_assignments?user.id=0e69b7f0c3574abc9de5fb580d22152a&scope.project.id=ec03e112e3d547fbab6faf7b485c3fca&role.id=a94bdb4c5796428baf63644c062dfcb2 => generated 259 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:20:45.682330 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd155d2c-441f-41cc-b267-053c09af5b11 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:45.726431 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d63c0a05-5b97-46e0-9404-ac5cc0dee1ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:45.727071 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d63c0a05-5b97-46e0-9404-ac5cc0dee1ae None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:45.727347 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d63c0a05-5b97-46e0-9404-ac5cc0dee1ae None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:45.727621 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d63c0a05-5b97-46e0-9404-ac5cc0dee1ae None admin] PATH_INFO: `/v3/roles/service` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:45.739208 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d63c0a05-5b97-46e0-9404-ac5cc0dee1ae None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 21:20:45.744547 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-d63c0a05-5b97-46e0-9404-ac5cc0dee1ae None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 29 21:20:45.744547 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:45.744547 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:45.744547 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:45.744547 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:45.744547 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:45.744547 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:45.744547 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:45.744547 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:45.744547 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:45.744547 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:45.744547 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:45.744547 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 21:20:45.744547 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 21:20:45.744547 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 21:20:45.744547 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 21:20:45.745459 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 21:20:45.745459 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 21:20:45.745459 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 21:20:45.745459 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 21:20:45.745459 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:45.745459 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:45.745459 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:45.745459 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:45.745459 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:45.745459 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:45.745459 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:45.745459 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:45.745459 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:45.745459 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:45.745459 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:45.745459 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:45.746292 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:45.746292 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:45.746292 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:45.746292 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:45.746292 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 21:20:45.746292 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 21:20:45.746292 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 21:20:45.746292 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 21:20:45.746292 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 21:20:45.746292 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 21:20:45.746292 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 29 21:20:45.746292 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:20:45.746292 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 274/544] 10.222.0.90 () {56 vars in 1102 bytes} [Mon Aug 29 21:20:45 2022] GET /identity/v3/roles/service => generated 85 bytes in 282 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 29 21:20:45.752662 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b419b528-a9fa-4b1e-9f0b-00b3c980f673 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:45.887818 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b419b528-a9fa-4b1e-9f0b-00b3c980f673 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:45.889507 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:45.933198 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02ab8aa4-a7c9-47c0-a45b-682f2cd91cb3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:45.933846 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02ab8aa4-a7c9-47c0-a45b-682f2cd91cb3 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:45.934129 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02ab8aa4-a7c9-47c0-a45b-682f2cd91cb3 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:45.934397 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02ab8aa4-a7c9-47c0-a45b-682f2cd91cb3 None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:45.935110 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02ab8aa4-a7c9-47c0-a45b-682f2cd91cb3 None admin] RBAC: Adding query filter params (name=service) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:45.935408 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02ab8aa4-a7c9-47c0-a45b-682f2cd91cb3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:45.936978 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02ab8aa4-a7c9-47c0-a45b-682f2cd91cb3 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:45.946429 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 271/545] 10.222.0.90 () {56 vars in 1103 bytes} [Mon Aug 29 21:20:45 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:20:45.954945 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d63c0a05-5b97-46e0-9404-ac5cc0dee1ae None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:45.991316 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:46.020236 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0b15503-e635-413f-95f5-00235c6f6978 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:46.020887 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0b15503-e635-413f-95f5-00235c6f6978 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:46.021370 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0b15503-e635-413f-95f5-00235c6f6978 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:46.023238 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 272/546] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:20:46 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 29 21:20:46.031398 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8379bdff-191c-4637-af9d-423c2e777572 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:46.031684 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8379bdff-191c-4637-af9d-423c2e777572 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:46.031963 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8379bdff-191c-4637-af9d-423c2e777572 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:46.056417 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-8379bdff-191c-4637-af9d-423c2e777572 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:20:46.057642 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-8379bdff-191c-4637-af9d-423c2e777572 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:20:46.123439 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d63c0a05-5b97-46e0-9404-ac5cc0dee1ae None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:46.177680 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 273/547] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:20:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 147 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 29 21:20:46.187897 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5665070e-4df6-41a7-8ebf-ec883a657dbf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:46.188473 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5665070e-4df6-41a7-8ebf-ec883a657dbf None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:46.188754 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5665070e-4df6-41a7-8ebf-ec883a657dbf None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:46.189025 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5665070e-4df6-41a7-8ebf-ec883a657dbf None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:46.200975 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5665070e-4df6-41a7-8ebf-ec883a657dbf None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:46.202490 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5665070e-4df6-41a7-8ebf-ec883a657dbf None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:46.223741 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-5665070e-4df6-41a7-8ebf-ec883a657dbf None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:20:46.223741 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:46.223741 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:46.223741 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:46.223741 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:46.223741 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:46.223741 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:46.223741 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:46.223741 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:46.223741 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:46.223741 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:46.223741 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:46.223741 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 21:20:46.223741 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 21:20:46.223741 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 21:20:46.223741 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 21:20:46.224688 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:46.224688 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:46.224688 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:46.224688 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:46.224688 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:46.224688 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:46.224688 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:46.224688 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:46.224688 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:46.224688 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:46.224688 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:46.224688 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:46.224688 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:46.224688 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:46.224688 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:46.224688 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:46.224688 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 21:20:46.225545 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 21:20:46.225545 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:20:46.225545 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:20:46.225545 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 275/548] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:20:45 2022] GET /identity/v3/domains/Default => generated 87 bytes in 270 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:20:46.232021 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8379bdff-191c-4637-af9d-423c2e777572 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:46.357186 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:46.369889 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8379bdff-191c-4637-af9d-423c2e777572 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:46.413352 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beea9871-cf75-4824-9fca-a8da3b619edd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:46.414000 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beea9871-cf75-4824-9fca-a8da3b619edd None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:46.414276 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beea9871-cf75-4824-9fca-a8da3b619edd None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:46.414545 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beea9871-cf75-4824-9fca-a8da3b619edd None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:46.415187 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beea9871-cf75-4824-9fca-a8da3b619edd None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:46.415487 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beea9871-cf75-4824-9fca-a8da3b619edd None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:46.416843 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beea9871-cf75-4824-9fca-a8da3b619edd None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:46.429120 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 274/549] 10.222.0.90 () {56 vars in 1109 bytes} [Mon Aug 29 21:20:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:46.435627 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5665070e-4df6-41a7-8ebf-ec883a657dbf None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:46.531848 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:46.558555 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5665070e-4df6-41a7-8ebf-ec883a657dbf None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:46.593912 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f019bc1b-5613-40d3-b3c5-b3a4fa4f1caf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:46.594554 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f019bc1b-5613-40d3-b3c5-b3a4fa4f1caf None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:46.594839 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f019bc1b-5613-40d3-b3c5-b3a4fa4f1caf None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:46.595176 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f019bc1b-5613-40d3-b3c5-b3a4fa4f1caf None admin] PATH_INFO: `/v3/users/swift` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:46.603308 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f019bc1b-5613-40d3-b3c5-b3a4fa4f1caf None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:46.604858 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f019bc1b-5613-40d3-b3c5-b3a4fa4f1caf None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:46.614784 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beea9871-cf75-4824-9fca-a8da3b619edd None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:46.616959 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-f019bc1b-5613-40d3-b3c5-b3a4fa4f1caf None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Aug 29 21:20:46.616959 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:46.616959 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:46.616959 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:46.616959 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:46.616959 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:46.616959 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:46.616959 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:46.616959 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:46.616959 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:46.616959 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:46.616959 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:46.616959 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 29 21:20:46.616959 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 29 21:20:46.616959 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 29 21:20:46.616959 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 29 21:20:46.618524 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:46.618524 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:46.618524 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 29 21:20:46.618524 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 21:20:46.618524 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 29 21:20:46.618524 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 21:20:46.618524 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:46.618524 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:46.618524 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:46.618524 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:46.618524 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:46.618524 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:46.618524 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:46.618524 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:46.618524 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:46.618524 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:46.618524 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:46.618524 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:46.619877 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:46.619877 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:46.619877 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 29 21:20:46.619877 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 29 21:20:46.619877 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 29 21:20:46.619877 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 29 21:20:46.619877 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 29 21:20:46.619877 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 29 21:20:46.619877 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Aug 29 21:20:46.619877 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:20:46.623100 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 276/550] 10.222.0.90 () {56 vars in 1096 bytes} [Mon Aug 29 21:20:46 2022] GET /identity/v3/users/swift => generated 83 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:20:46.628671 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f019bc1b-5613-40d3-b3c5-b3a4fa4f1caf None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:46.740824 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f019bc1b-5613-40d3-b3c5-b3a4fa4f1caf None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:46.775939 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f83fadd2-6d3b-4dfb-84ab-a3afd4bfd1aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:46.776564 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f83fadd2-6d3b-4dfb-84ab-a3afd4bfd1aa None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:46.776850 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f83fadd2-6d3b-4dfb-84ab-a3afd4bfd1aa None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:46.777126 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f83fadd2-6d3b-4dfb-84ab-a3afd4bfd1aa None admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:46.777889 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f83fadd2-6d3b-4dfb-84ab-a3afd4bfd1aa None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:46.778202 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f83fadd2-6d3b-4dfb-84ab-a3afd4bfd1aa None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:46.779633 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f83fadd2-6d3b-4dfb-84ab-a3afd4bfd1aa None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:46.787756 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:46.800709 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 277/551] 10.222.0.90 () {56 vars in 1135 bytes} [Mon Aug 29 21:20:46 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:20:46.806221 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f83fadd2-6d3b-4dfb-84ab-a3afd4bfd1aa None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:46.860178 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beea9871-cf75-4824-9fca-a8da3b619edd None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:46.926295 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f83fadd2-6d3b-4dfb-84ab-a3afd4bfd1aa None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:46.963207 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d7e808b-1729-40b6-b164-c68f8d448b04 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:46.964108 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d7e808b-1729-40b6-b164-c68f8d448b04 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:46.964542 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d7e808b-1729-40b6-b164-c68f8d448b04 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:46.964963 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d7e808b-1729-40b6-b164-c68f8d448b04 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:46.999103 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7699d2b0-76d6-4247-af84-7ee252fad505 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:46.999769 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7699d2b0-76d6-4247-af84-7ee252fad505 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:47.000052 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7699d2b0-76d6-4247-af84-7ee252fad505 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:47.000323 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7699d2b0-76d6-4247-af84-7ee252fad505 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:47.034753 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7699d2b0-76d6-4247-af84-7ee252fad505 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 21:20:47.040041 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:47.040351 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-7699d2b0-76d6-4247-af84-7ee252fad505 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 29 21:20:47.040351 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:47.040351 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:47.040351 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:47.040351 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:47.040351 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:47.040351 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:47.040351 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:47.040351 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:47.040351 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:47.040351 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:47.040351 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:47.040351 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 21:20:47.040351 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 21:20:47.040351 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 21:20:47.040351 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 21:20:47.041771 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 21:20:47.041771 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 21:20:47.041771 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 21:20:47.041771 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 21:20:47.041771 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:47.041771 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:47.041771 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:47.041771 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:47.041771 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:47.041771 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:47.041771 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:47.041771 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:47.041771 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:47.041771 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:47.041771 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:47.041771 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:47.042860 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:47.042860 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:47.042860 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:47.042860 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:47.042860 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 21:20:47.042860 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 21:20:47.042860 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 21:20:47.042860 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 21:20:47.042860 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 21:20:47.042860 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 21:20:47.042860 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 29 21:20:47.042860 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:20:47.042860 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 275/552] 10.222.0.90 () {56 vars in 1102 bytes} [Mon Aug 29 21:20:46 2022] GET /identity/v3/roles/service => generated 85 bytes in 428 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 29 21:20:47.044815 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d7e808b-1729-40b6-b164-c68f8d448b04 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:47.048138 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d7e808b-1729-40b6-b164-c68f8d448b04 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:47.049156 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7699d2b0-76d6-4247-af84-7ee252fad505 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:47.069842 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:47.157176 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 278/553] 10.222.0.90 () {58 vars in 1282 bytes} [Mon Aug 29 21:20:46 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 351 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Aug 29 21:20:47.162839 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d7e808b-1729-40b6-b164-c68f8d448b04 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:47.253548 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7699d2b0-76d6-4247-af84-7ee252fad505 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:47.317929 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:47.322188 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91bc4cfb-cc0e-4127-84d8-c2e5796890c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:47.323004 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91bc4cfb-cc0e-4127-84d8-c2e5796890c9 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:47.323365 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91bc4cfb-cc0e-4127-84d8-c2e5796890c9 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:47.323762 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91bc4cfb-cc0e-4127-84d8-c2e5796890c9 None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:47.324408 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d7e808b-1729-40b6-b164-c68f8d448b04 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:47.326516 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91bc4cfb-cc0e-4127-84d8-c2e5796890c9 None admin] RBAC: Adding query filter params (name=service) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:47.327860 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91bc4cfb-cc0e-4127-84d8-c2e5796890c9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:47.329252 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91bc4cfb-cc0e-4127-84d8-c2e5796890c9 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:47.347742 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 276/554] 10.222.0.90 () {56 vars in 1103 bytes} [Mon Aug 29 21:20:47 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 299 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:47.353776 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91bc4cfb-cc0e-4127-84d8-c2e5796890c9 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:47.397043 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56ff7aae-2eb2-4a14-9fb4-c2cc2383171c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:47.397700 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ff7aae-2eb2-4a14-9fb4-c2cc2383171c None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:47.398014 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ff7aae-2eb2-4a14-9fb4-c2cc2383171c None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:47.398290 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ff7aae-2eb2-4a14-9fb4-c2cc2383171c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:47.417974 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56ff7aae-2eb2-4a14-9fb4-c2cc2383171c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:47.419448 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56ff7aae-2eb2-4a14-9fb4-c2cc2383171c None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:47.438059 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-56ff7aae-2eb2-4a14-9fb4-c2cc2383171c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:20:47.438059 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:47.438059 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:47.438059 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:47.438059 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:47.438059 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:47.438059 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:47.438059 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:47.438059 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:47.438059 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:47.438059 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:47.438059 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:47.438059 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 21:20:47.438059 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 21:20:47.438059 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 21:20:47.438059 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 21:20:47.439064 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:47.439064 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:47.439064 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:47.439064 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:47.439064 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:47.439064 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:47.439064 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:47.439064 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:47.439064 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:47.439064 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:47.439064 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:47.439064 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:47.439064 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:47.439064 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:47.439064 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:47.439064 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:47.439064 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 21:20:47.439967 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 21:20:47.439967 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:20:47.439967 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:20:47.439967 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 279/555] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:20:47 2022] GET /identity/v3/domains/Default => generated 87 bytes in 277 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 29 21:20:47.445452 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56ff7aae-2eb2-4a14-9fb4-c2cc2383171c None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:47.461176 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:47.496886 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91bc4cfb-cc0e-4127-84d8-c2e5796890c9 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:47.533646 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cfbd1ae-5a95-4d33-bb77-295dfe9591ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:47.535293 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cfbd1ae-5a95-4d33-bb77-295dfe9591ce None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:47.535915 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cfbd1ae-5a95-4d33-bb77-295dfe9591ce None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:47.536347 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cfbd1ae-5a95-4d33-bb77-295dfe9591ce None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:47.548638 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cfbd1ae-5a95-4d33-bb77-295dfe9591ce None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:47.551116 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cfbd1ae-5a95-4d33-bb77-295dfe9591ce None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:47.567820 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56ff7aae-2eb2-4a14-9fb4-c2cc2383171c None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:47.568536 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-9cfbd1ae-5a95-4d33-bb77-295dfe9591ce None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:20:47.568536 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:47.568536 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:47.568536 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:47.568536 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:47.568536 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:47.568536 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:47.568536 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:47.568536 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:47.568536 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:47.568536 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:47.568536 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:47.568536 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 21:20:47.568536 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 21:20:47.568536 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 21:20:47.568536 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 21:20:47.570011 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:47.570011 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:47.570011 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:47.570011 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:47.570011 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:47.570011 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:47.570011 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:47.570011 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:47.570011 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:47.570011 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:47.570011 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:47.570011 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:47.570011 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:47.570011 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:47.570011 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:47.570011 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:47.570011 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 21:20:47.570930 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 21:20:47.570930 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:20:47.570930 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:20:47.574265 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 277/556] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:20:47 2022] GET /identity/v3/domains/Default => generated 87 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:20:47.575872 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-717d973a-08de-4bcb-80e5-ace8ccc93f36 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:47.576380 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-717d973a-08de-4bcb-80e5-ace8ccc93f36 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:47.576865 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-717d973a-08de-4bcb-80e5-ace8ccc93f36 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:47.578444 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 278/557] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:20:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:20:47.579434 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-717d973a-08de-4bcb-80e5-ace8ccc93f36 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:47.610238 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-287bd425-f124-4f26-a92f-708d38bd7820 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:47.611158 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-287bd425-f124-4f26-a92f-708d38bd7820 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:47.611630 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-287bd425-f124-4f26-a92f-708d38bd7820 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:47.612181 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-287bd425-f124-4f26-a92f-708d38bd7820 None admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:47.613111 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-287bd425-f124-4f26-a92f-708d38bd7820 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:47.613565 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-287bd425-f124-4f26-a92f-708d38bd7820 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:47.615117 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-287bd425-f124-4f26-a92f-708d38bd7820 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:47.633923 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 280/558] 10.222.0.90 () {56 vars in 1109 bytes} [Mon Aug 29 21:20:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:20:47.635614 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f72d6fb-65f7-448f-a1c0-f493c03f7e4f None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:47.636054 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f72d6fb-65f7-448f-a1c0-f493c03f7e4f None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:47.636470 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f72d6fb-65f7-448f-a1c0-f493c03f7e4f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:47.644878 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:47.656701 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-2f72d6fb-65f7-448f-a1c0-f493c03f7e4f None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:20:47.657963 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-2f72d6fb-65f7-448f-a1c0-f493c03f7e4f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:20:47.693045 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-717d973a-08de-4bcb-80e5-ace8ccc93f36 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:47.717902 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:47.730828 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cf9fa7b-ff62-4bc8-9deb-77c4772c9a17 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:47.731440 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cf9fa7b-ff62-4bc8-9deb-77c4772c9a17 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:47.731718 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cf9fa7b-ff62-4bc8-9deb-77c4772c9a17 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:47.731989 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cf9fa7b-ff62-4bc8-9deb-77c4772c9a17 None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:47.732630 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cf9fa7b-ff62-4bc8-9deb-77c4772c9a17 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:47.732930 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cf9fa7b-ff62-4bc8-9deb-77c4772c9a17 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:47.733449 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 281/559] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:20:47 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 21:20:47.734384 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cf9fa7b-ff62-4bc8-9deb-77c4772c9a17 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:47.734738 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f72d6fb-65f7-448f-a1c0-f493c03f7e4f None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:47.759159 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 279/560] 10.222.0.90 () {56 vars in 1109 bytes} [Mon Aug 29 21:20:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:47.760687 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ad56e48-1355-4014-aea3-5b0b0f9cb531 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:47.761135 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ad56e48-1355-4014-aea3-5b0b0f9cb531 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:47.761402 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ad56e48-1355-4014-aea3-5b0b0f9cb531 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:47.762697 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 280/561] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:20:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:20:47.763517 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ad56e48-1355-4014-aea3-5b0b0f9cb531 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:47.873503 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f72d6fb-65f7-448f-a1c0-f493c03f7e4f None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:47.875567 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ad56e48-1355-4014-aea3-5b0b0f9cb531 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:47.911043 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f35dccd9-5163-4237-b6af-6d073aa0c843 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:47.911919 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f35dccd9-5163-4237-b6af-6d073aa0c843 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:47.912288 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f35dccd9-5163-4237-b6af-6d073aa0c843 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:47.912564 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f35dccd9-5163-4237-b6af-6d073aa0c843 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:47.914629 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:47.915100 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c582a9ea-5cf6-40b7-9b3c-72a903f60379 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:47.915890 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c582a9ea-5cf6-40b7-9b3c-72a903f60379 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:47.916166 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c582a9ea-5cf6-40b7-9b3c-72a903f60379 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:47.916445 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c582a9ea-5cf6-40b7-9b3c-72a903f60379 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:47.919685 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f35dccd9-5163-4237-b6af-6d073aa0c843 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:47.921347 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f35dccd9-5163-4237-b6af-6d073aa0c843 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:47.924258 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c582a9ea-5cf6-40b7-9b3c-72a903f60379 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:47.926324 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c582a9ea-5cf6-40b7-9b3c-72a903f60379 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:47.933284 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-f35dccd9-5163-4237-b6af-6d073aa0c843 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 21:20:47.933284 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:47.933284 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:47.933284 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:47.933284 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:47.933284 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:47.933284 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:47.933284 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:47.933284 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:47.933284 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:47.933284 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:47.933284 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:47.933284 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 29 21:20:47.933284 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 29 21:20:47.933284 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 29 21:20:47.934454 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 29 21:20:47.934454 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:47.934454 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:47.934454 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:47.934454 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:47.934454 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:47.934454 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:47.934454 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:47.934454 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:47.934454 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:47.934454 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:47.934454 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:47.934454 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:47.934454 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:47.934454 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:47.934454 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:47.934454 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:47.935439 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 29 21:20:47.935439 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 29 21:20:47.935439 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 29 21:20:47.935439 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 29 21:20:47.935439 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 29 21:20:47.935439 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 29 21:20:47.935439 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 21:20:47.935439 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:20:47.938031 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-c582a9ea-5cf6-40b7-9b3c-72a903f60379 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 29 21:20:47.938031 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:47.938031 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:47.938031 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:47.938031 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:47.938031 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:47.938031 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:47.938031 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:47.938031 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:47.938031 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:47.938031 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:47.938031 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:47.938031 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 29 21:20:47.938031 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 29 21:20:47.938031 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 29 21:20:47.938031 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 29 21:20:47.939417 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:47.939417 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:47.939417 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 29 21:20:47.939417 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 21:20:47.939417 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 29 21:20:47.939417 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 21:20:47.939417 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:47.939417 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:47.939417 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:47.939417 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:47.939417 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:47.939417 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:47.939417 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:47.939417 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:47.939417 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:47.939417 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:47.939417 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:47.939417 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:47.940428 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:47.940428 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:47.940428 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 29 21:20:47.940428 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 29 21:20:47.940428 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 29 21:20:47.940428 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 29 21:20:47.940428 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 29 21:20:47.940428 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 29 21:20:47.940428 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 29 21:20:47.940428 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:20:47.942176 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 282/562] 10.222.0.90 () {56 vars in 1111 bytes} [Mon Aug 29 21:20:47 2022] GET /identity/v3/projects/service => generated 88 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:20:47.942324 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 281/563] 10.222.0.90 () {56 vars in 1093 bytes} [Mon Aug 29 21:20:47 2022] GET /identity/v3/users/nova => generated 82 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:20:47.942465 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43fcae11-8399-4f5a-8061-5f13c5c0dfe3 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:47.942465 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43fcae11-8399-4f5a-8061-5f13c5c0dfe3 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:47.942465 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43fcae11-8399-4f5a-8061-5f13c5c0dfe3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:47.942883 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14ffcdcb-4315-44b0-a301-167cc05e9ee3 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:47.942883 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14ffcdcb-4315-44b0-a301-167cc05e9ee3 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:47.942883 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14ffcdcb-4315-44b0-a301-167cc05e9ee3 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:47.947157 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 282/564] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:20:47 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:20:47.947909 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14ffcdcb-4315-44b0-a301-167cc05e9ee3 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:47.964821 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-43fcae11-8399-4f5a-8061-5f13c5c0dfe3 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:20:47.966041 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-43fcae11-8399-4f5a-8061-5f13c5c0dfe3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:20:47.997743 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:48.033338 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 283/565] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:20:47 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 21:20:48.034847 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43fcae11-8399-4f5a-8061-5f13c5c0dfe3 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:48.059783 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14ffcdcb-4315-44b0-a301-167cc05e9ee3 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:48.100183 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f2b0e6b-0a67-4216-a2d2-4babee1aa2dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:48.100804 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f2b0e6b-0a67-4216-a2d2-4babee1aa2dc None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:48.101080 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f2b0e6b-0a67-4216-a2d2-4babee1aa2dc None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:48.101348 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f2b0e6b-0a67-4216-a2d2-4babee1aa2dc None admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:48.102062 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f2b0e6b-0a67-4216-a2d2-4babee1aa2dc None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:48.102367 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f2b0e6b-0a67-4216-a2d2-4babee1aa2dc None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:48.103798 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f2b0e6b-0a67-4216-a2d2-4babee1aa2dc None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:48.116993 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 283/566] 10.222.0.90 () {56 vars in 1148 bytes} [Mon Aug 29 21:20:47 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:48.118130 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6a4c259-9d54-4447-9f02-7990b7d79c62 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:48.118412 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6a4c259-9d54-4447-9f02-7990b7d79c62 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:48.118681 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6a4c259-9d54-4447-9f02-7990b7d79c62 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:48.136310 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-d6a4c259-9d54-4447-9f02-7990b7d79c62 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:20:48.137571 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-d6a4c259-9d54-4447-9f02-7990b7d79c62 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:20:48.149117 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43fcae11-8399-4f5a-8061-5f13c5c0dfe3 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:48.174037 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:48.188425 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12bd0601-9042-416c-ad0d-df10edf54fce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:48.189134 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12bd0601-9042-416c-ad0d-df10edf54fce None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:48.189488 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12bd0601-9042-416c-ad0d-df10edf54fce None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:48.189866 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12bd0601-9042-416c-ad0d-df10edf54fce None admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:48.190649 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12bd0601-9042-416c-ad0d-df10edf54fce None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:48.191032 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12bd0601-9042-416c-ad0d-df10edf54fce None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:48.192463 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12bd0601-9042-416c-ad0d-df10edf54fce None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:48.208449 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 284/567] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:20:48 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 21:20:48.209284 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6a4c259-9d54-4447-9f02-7990b7d79c62 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:48.223526 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 284/568] 10.222.0.90 () {56 vars in 1133 bytes} [Mon Aug 29 21:20:48 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:48.223526 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12bd0601-9042-416c-ad0d-df10edf54fce None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:48.247786 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:48.324906 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6a4c259-9d54-4447-9f02-7990b7d79c62 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:48.345317 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12bd0601-9042-416c-ad0d-df10edf54fce None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:48.364498 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc134703-bbb0-4306-b98a-1f08b08e48d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:48.365138 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc134703-bbb0-4306-b98a-1f08b08e48d3 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:48.365414 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc134703-bbb0-4306-b98a-1f08b08e48d3 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:48.365685 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc134703-bbb0-4306-b98a-1f08b08e48d3 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:48.376831 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc134703-bbb0-4306-b98a-1f08b08e48d3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 21:20:48.380586 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2059a533-3d5e-4612-a5e9-0bdcb0ee6d9e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:48.381197 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2059a533-3d5e-4612-a5e9-0bdcb0ee6d9e None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:48.381480 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2059a533-3d5e-4612-a5e9-0bdcb0ee6d9e None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:48.381753 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2059a533-3d5e-4612-a5e9-0bdcb0ee6d9e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:48.382236 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-fc134703-bbb0-4306-b98a-1f08b08e48d3 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 29 21:20:48.382236 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:48.382236 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:48.382236 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:48.382236 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:48.382236 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:48.382236 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:48.382236 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:48.382236 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:48.382236 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:48.382236 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:48.382236 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:48.382236 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 21:20:48.382236 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 21:20:48.382236 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 21:20:48.382236 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 21:20:48.383154 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 21:20:48.383154 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 21:20:48.383154 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 21:20:48.383154 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 21:20:48.383154 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:48.383154 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:48.383154 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:48.383154 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:48.383154 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:48.383154 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:48.383154 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:48.383154 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:48.383154 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:48.383154 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:48.383154 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:48.383154 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:48.384002 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:48.384002 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:48.384002 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:48.384002 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:48.384002 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 21:20:48.384002 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 21:20:48.384002 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 21:20:48.384002 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 21:20:48.384002 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 21:20:48.384002 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 21:20:48.384002 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 29 21:20:48.384002 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:20:48.384672 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2059a533-3d5e-4612-a5e9-0bdcb0ee6d9e None admin] RBAC: Adding query filter params (role.id=a94bdb4c5796428baf63644c062dfcb2, scope.project.id=ec03e112e3d547fbab6faf7b485c3fca, user.id=139bbbde6d884b42b5440727926e2c58) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:48.384672 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2059a533-3d5e-4612-a5e9-0bdcb0ee6d9e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:48.384798 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 285/569] 10.222.0.90 () {56 vars in 1102 bytes} [Mon Aug 29 21:20:48 2022] GET /identity/v3/roles/service => generated 85 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:20:48.384798 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc134703-bbb0-4306-b98a-1f08b08e48d3 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:48.384938 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2059a533-3d5e-4612-a5e9-0bdcb0ee6d9e None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:48.397955 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 285/570] 10.222.0.90 () {56 vars in 1374 bytes} [Mon Aug 29 21:20:48 2022] GET /identity/v3/role_assignments?user.id=139bbbde6d884b42b5440727926e2c58&scope.project.id=ec03e112e3d547fbab6faf7b485c3fca&role.id=a94bdb4c5796428baf63644c062dfcb2 => generated 259 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:48.397955 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2059a533-3d5e-4612-a5e9-0bdcb0ee6d9e None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:48.462590 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:48.519386 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:48.527354 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2059a533-3d5e-4612-a5e9-0bdcb0ee6d9e None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:48.531505 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc134703-bbb0-4306-b98a-1f08b08e48d3 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:48.561944 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96a61668-9121-4501-bdee-ec50d59bf863 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:48.562587 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96a61668-9121-4501-bdee-ec50d59bf863 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:48.562872 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96a61668-9121-4501-bdee-ec50d59bf863 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:48.563149 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96a61668-9121-4501-bdee-ec50d59bf863 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:48.574656 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96a61668-9121-4501-bdee-ec50d59bf863 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 21:20:48.578125 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93b6885d-394c-4de7-99fa-190b5cce6ad0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:48.578764 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93b6885d-394c-4de7-99fa-190b5cce6ad0 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:48.579046 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93b6885d-394c-4de7-99fa-190b5cce6ad0 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:48.579319 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93b6885d-394c-4de7-99fa-190b5cce6ad0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:48.580013 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-96a61668-9121-4501-bdee-ec50d59bf863 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 29 21:20:48.580013 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:48.580013 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:48.580013 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:48.580013 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:48.580013 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:48.580013 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:48.580013 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:48.580013 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:48.580013 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:48.580013 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:48.580013 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:48.580013 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 21:20:48.580013 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 21:20:48.580013 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 21:20:48.580013 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 21:20:48.580937 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 21:20:48.580937 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 21:20:48.580937 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 21:20:48.580937 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 21:20:48.580937 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:48.580937 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:48.580937 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:48.580937 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:48.580937 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:48.580937 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:48.580937 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:48.580937 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:48.580937 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:48.580937 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:48.580937 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:48.580937 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:48.582507 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:48.582507 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:48.582507 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:48.582507 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:48.582507 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 21:20:48.582507 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 21:20:48.582507 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 21:20:48.582507 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 21:20:48.582507 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 21:20:48.582507 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 21:20:48.582507 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 29 21:20:48.582507 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:20:48.586704 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 286/571] 10.222.0.90 () {56 vars in 1102 bytes} [Mon Aug 29 21:20:48 2022] GET /identity/v3/roles/service => generated 85 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:20:48.587454 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96a61668-9121-4501-bdee-ec50d59bf863 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:48.593877 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93b6885d-394c-4de7-99fa-190b5cce6ad0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:48.595437 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93b6885d-394c-4de7-99fa-190b5cce6ad0 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:48.611147 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-93b6885d-394c-4de7-99fa-190b5cce6ad0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:20:48.611147 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:48.611147 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:48.611147 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:48.611147 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:48.611147 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:48.611147 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:48.611147 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:48.611147 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:48.611147 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:48.611147 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:48.611147 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:48.611147 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 21:20:48.611147 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 21:20:48.611147 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 21:20:48.611147 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 21:20:48.612166 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:48.612166 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:48.612166 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:48.612166 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:48.612166 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:48.612166 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:48.612166 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:48.612166 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:48.612166 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:48.612166 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:48.612166 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:48.612166 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:48.612166 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:48.612166 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:48.612166 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:48.612166 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:48.612166 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 21:20:48.613179 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 21:20:48.613179 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:20:48.613179 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:20:48.613179 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 286/572] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:20:48 2022] GET /identity/v3/domains/Default => generated 87 bytes in 228 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:20:48.613416 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93b6885d-394c-4de7-99fa-190b5cce6ad0 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:48.703711 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96a61668-9121-4501-bdee-ec50d59bf863 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:48.711575 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:48.721666 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93b6885d-394c-4de7-99fa-190b5cce6ad0 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:48.742148 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5a123c7-f59b-42cd-9abd-3ffc3d9906dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:48.742868 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5a123c7-f59b-42cd-9abd-3ffc3d9906dd None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:48.743619 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5a123c7-f59b-42cd-9abd-3ffc3d9906dd None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:48.743619 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5a123c7-f59b-42cd-9abd-3ffc3d9906dd None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:48.744309 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5a123c7-f59b-42cd-9abd-3ffc3d9906dd None admin] RBAC: Adding query filter params (name=service) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:48.744689 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5a123c7-f59b-42cd-9abd-3ffc3d9906dd None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:48.746167 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5a123c7-f59b-42cd-9abd-3ffc3d9906dd None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:48.758709 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb34db8c-554a-40f9-97a5-4998cff42f0c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:48.759236 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 287/573] 10.222.0.90 () {56 vars in 1103 bytes} [Mon Aug 29 21:20:48 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:48.759760 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb34db8c-554a-40f9-97a5-4998cff42f0c None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:48.760108 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb34db8c-554a-40f9-97a5-4998cff42f0c None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:48.760223 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5a123c7-f59b-42cd-9abd-3ffc3d9906dd None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:48.760608 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb34db8c-554a-40f9-97a5-4998cff42f0c None admin] PATH_INFO: `/v3/roles/service` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:48.772019 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb34db8c-554a-40f9-97a5-4998cff42f0c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 21:20:48.777374 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-fb34db8c-554a-40f9-97a5-4998cff42f0c None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 29 21:20:48.777374 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:48.777374 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:48.777374 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:48.777374 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:48.777374 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:48.777374 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:48.777374 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:48.777374 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:48.777374 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:48.777374 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:48.777374 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:48.777374 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 21:20:48.777374 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 21:20:48.777374 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 21:20:48.777374 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 21:20:48.778600 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 21:20:48.778600 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 21:20:48.778600 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 21:20:48.778600 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 21:20:48.778600 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:48.778600 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:48.778600 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:48.778600 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:48.778600 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:48.778600 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:48.778600 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:48.778600 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:48.778600 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:48.778600 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:48.778600 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:48.778600 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:48.782290 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:48.782290 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:48.782290 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:48.782290 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:48.782290 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 21:20:48.782290 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 21:20:48.782290 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 21:20:48.782290 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 21:20:48.782290 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 21:20:48.782290 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 21:20:48.782290 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 29 21:20:48.782290 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:20:48.782290 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 287/574] 10.222.0.90 () {56 vars in 1102 bytes} [Mon Aug 29 21:20:48 2022] GET /identity/v3/roles/service => generated 85 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:20:48.782290 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb34db8c-554a-40f9-97a5-4998cff42f0c None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:48.788504 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:48.880512 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5a123c7-f59b-42cd-9abd-3ffc3d9906dd None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:48.897675 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb34db8c-554a-40f9-97a5-4998cff42f0c None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:48.917842 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba541df5-eaef-4c01-8669-053cbebda57c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:48.918660 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba541df5-eaef-4c01-8669-053cbebda57c None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:48.919006 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba541df5-eaef-4c01-8669-053cbebda57c None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:48.919337 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba541df5-eaef-4c01-8669-053cbebda57c None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:48.920064 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba541df5-eaef-4c01-8669-053cbebda57c None admin] RBAC: Adding query filter params (name=service) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:48.920439 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba541df5-eaef-4c01-8669-053cbebda57c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:48.922293 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba541df5-eaef-4c01-8669-053cbebda57c None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:48.933454 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 288/575] 10.222.0.90 () {56 vars in 1103 bytes} [Mon Aug 29 21:20:48 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:48.934442 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba541df5-eaef-4c01-8669-053cbebda57c None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:48.938378 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-140abef7-ac51-4ea9-a7fe-62a258a8f2d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:48.939075 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-140abef7-ac51-4ea9-a7fe-62a258a8f2d5 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:48.939413 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-140abef7-ac51-4ea9-a7fe-62a258a8f2d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:48.939751 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-140abef7-ac51-4ea9-a7fe-62a258a8f2d5 None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:48.941098 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-140abef7-ac51-4ea9-a7fe-62a258a8f2d5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:48.942105 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-140abef7-ac51-4ea9-a7fe-62a258a8f2d5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:48.943396 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-140abef7-ac51-4ea9-a7fe-62a258a8f2d5 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:48.960267 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 288/576] 10.222.0.90 () {56 vars in 1109 bytes} [Mon Aug 29 21:20:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:48.960267 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-140abef7-ac51-4ea9-a7fe-62a258a8f2d5 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:48.975305 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:49.030674 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:49.051991 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba541df5-eaef-4c01-8669-053cbebda57c None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:49.070344 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-140abef7-ac51-4ea9-a7fe-62a258a8f2d5 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:49.088714 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7af59ce6-4fd3-4b62-bc79-e41a8eceb2d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:49.089396 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7af59ce6-4fd3-4b62-bc79-e41a8eceb2d1 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:49.089760 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7af59ce6-4fd3-4b62-bc79-e41a8eceb2d1 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:49.090046 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7af59ce6-4fd3-4b62-bc79-e41a8eceb2d1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:49.102233 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7af59ce6-4fd3-4b62-bc79-e41a8eceb2d1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:49.103929 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7af59ce6-4fd3-4b62-bc79-e41a8eceb2d1 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:49.108071 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-203e5872-9893-48bc-aba7-a4efa36670b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:49.108781 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-203e5872-9893-48bc-aba7-a4efa36670b5 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:49.109160 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-203e5872-9893-48bc-aba7-a4efa36670b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:49.109547 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-203e5872-9893-48bc-aba7-a4efa36670b5 None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:49.110347 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-203e5872-9893-48bc-aba7-a4efa36670b5 None admin] RBAC: Adding query filter params (name=service) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:49.110765 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-203e5872-9893-48bc-aba7-a4efa36670b5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:49.112923 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-203e5872-9893-48bc-aba7-a4efa36670b5 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:49.119186 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-7af59ce6-4fd3-4b62-bc79-e41a8eceb2d1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:20:49.119186 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:49.119186 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:49.119186 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:49.119186 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:49.119186 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:49.119186 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:49.119186 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:49.119186 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:49.119186 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:49.119186 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:49.119186 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:49.119186 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 21:20:49.119186 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 21:20:49.119186 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 21:20:49.119186 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 21:20:49.124971 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:49.124971 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:49.124971 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:49.124971 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:49.124971 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:49.124971 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:49.124971 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:49.124971 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:49.124971 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:49.124971 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:49.124971 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:49.124971 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:49.124971 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:49.124971 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:49.124971 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:49.124971 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:49.124971 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 21:20:49.126173 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 21:20:49.126173 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:20:49.126173 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:20:49.126173 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 289/577] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:20:48 2022] GET /identity/v3/domains/Default => generated 87 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:20:49.126173 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7af59ce6-4fd3-4b62-bc79-e41a8eceb2d1 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:49.126481 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 289/578] 10.222.0.90 () {56 vars in 1103 bytes} [Mon Aug 29 21:20:48 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:49.126481 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-203e5872-9893-48bc-aba7-a4efa36670b5 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:49.222507 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:49.230265 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-203e5872-9893-48bc-aba7-a4efa36670b5 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:49.246147 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7af59ce6-4fd3-4b62-bc79-e41a8eceb2d1 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:49.257444 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:49.266621 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cf31730-4aa8-4cf5-a3a0-b9a30f21f353 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:49.267386 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cf31730-4aa8-4cf5-a3a0-b9a30f21f353 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:49.267746 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cf31730-4aa8-4cf5-a3a0-b9a30f21f353 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:49.268090 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cf31730-4aa8-4cf5-a3a0-b9a30f21f353 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:49.275984 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cf31730-4aa8-4cf5-a3a0-b9a30f21f353 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:49.277341 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cf31730-4aa8-4cf5-a3a0-b9a30f21f353 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:49.288874 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-7cf31730-4aa8-4cf5-a3a0-b9a30f21f353 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 21:20:49.288874 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:49.288874 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:49.288874 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:49.288874 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:49.288874 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:49.288874 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:49.288874 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:49.288874 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:49.288874 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:49.288874 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:49.288874 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:49.288874 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 29 21:20:49.288874 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 29 21:20:49.288874 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 29 21:20:49.290110 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 29 21:20:49.290110 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:49.290110 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:49.290110 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:49.290110 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:49.290110 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:49.290110 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:49.290110 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:49.290110 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:49.290110 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:49.290110 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:49.290110 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:49.290110 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:49.290110 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:49.290110 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:49.290110 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:49.290110 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:49.296562 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 29 21:20:49.296562 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 29 21:20:49.296562 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 29 21:20:49.296562 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 29 21:20:49.296562 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 29 21:20:49.296562 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 29 21:20:49.296562 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 21:20:49.296562 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:20:49.297113 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-234fdbae-97b0-4fe3-a18e-e34de1732475 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:49.297221 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 290/579] 10.222.0.90 () {56 vars in 1111 bytes} [Mon Aug 29 21:20:49 2022] GET /identity/v3/projects/service => generated 88 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:20:49.297316 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-234fdbae-97b0-4fe3-a18e-e34de1732475 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:49.297316 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-234fdbae-97b0-4fe3-a18e-e34de1732475 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:49.297482 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cf31730-4aa8-4cf5-a3a0-b9a30f21f353 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:49.297566 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-234fdbae-97b0-4fe3-a18e-e34de1732475 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:49.309274 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-234fdbae-97b0-4fe3-a18e-e34de1732475 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:49.311779 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-234fdbae-97b0-4fe3-a18e-e34de1732475 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:49.332584 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-234fdbae-97b0-4fe3-a18e-e34de1732475 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:20:49.332584 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:49.332584 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:49.332584 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:49.332584 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:49.332584 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:49.332584 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:49.332584 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:49.332584 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:49.332584 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:49.332584 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:49.332584 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:49.332584 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 21:20:49.332584 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 21:20:49.332584 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 21:20:49.332584 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 21:20:49.333574 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:49.333574 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:49.333574 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:49.333574 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:49.333574 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:49.333574 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:49.333574 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:49.333574 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:49.333574 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:49.333574 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:49.333574 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:49.333574 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:49.333574 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:49.333574 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:49.333574 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:49.333574 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:49.333574 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 21:20:49.334626 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 21:20:49.334626 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:20:49.334626 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:20:49.334626 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 290/580] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:20:49 2022] GET /identity/v3/domains/Default => generated 87 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:20:49.335285 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-234fdbae-97b0-4fe3-a18e-e34de1732475 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:49.402984 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cf31730-4aa8-4cf5-a3a0-b9a30f21f353 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:49.443092 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5d5914f-a69d-4823-ab6a-266557c2441b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:49.443717 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5d5914f-a69d-4823-ab6a-266557c2441b None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:49.444001 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5d5914f-a69d-4823-ab6a-266557c2441b None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:49.444274 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5d5914f-a69d-4823-ab6a-266557c2441b None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:49.444975 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5d5914f-a69d-4823-ab6a-266557c2441b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:49.445284 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5d5914f-a69d-4823-ab6a-266557c2441b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:49.446717 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5d5914f-a69d-4823-ab6a-266557c2441b None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:49.447639 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-234fdbae-97b0-4fe3-a18e-e34de1732475 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:49.459826 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 291/581] 10.222.0.90 () {56 vars in 1109 bytes} [Mon Aug 29 21:20:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:49.463282 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5d5914f-a69d-4823-ab6a-266557c2441b None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:49.484768 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee42255c-1313-4234-82f3-c442448baca7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:49.485366 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee42255c-1313-4234-82f3-c442448baca7 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:49.485616 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee42255c-1313-4234-82f3-c442448baca7 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:49.485978 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee42255c-1313-4234-82f3-c442448baca7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:49.488454 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:49.498782 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee42255c-1313-4234-82f3-c442448baca7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:49.500197 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee42255c-1313-4234-82f3-c442448baca7 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:49.516313 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:49.516463 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-ee42255c-1313-4234-82f3-c442448baca7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:20:49.516463 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:49.516463 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:49.516463 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:49.516463 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:49.516463 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:49.516463 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:49.516463 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:49.516463 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:49.516463 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:49.516463 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:49.516463 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:49.516463 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 21:20:49.516463 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 21:20:49.516463 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 21:20:49.516463 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 21:20:49.517398 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:49.517398 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:49.517398 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:49.517398 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:49.517398 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:49.517398 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:49.517398 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:49.517398 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:49.517398 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:49.517398 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:49.517398 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:49.517398 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:49.517398 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:49.517398 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:49.517398 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:49.517398 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:49.517398 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 21:20:49.518327 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 21:20:49.518327 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:20:49.518327 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:20:49.518327 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 291/582] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:20:49 2022] GET /identity/v3/domains/Default => generated 87 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:20:49.519787 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee42255c-1313-4234-82f3-c442448baca7 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:49.579568 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5d5914f-a69d-4823-ab6a-266557c2441b None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:49.616185 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bfb4827-a746-4ea7-898a-a9c8858ee645 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:49.616798 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bfb4827-a746-4ea7-898a-a9c8858ee645 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:49.617079 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bfb4827-a746-4ea7-898a-a9c8858ee645 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:49.617349 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bfb4827-a746-4ea7-898a-a9c8858ee645 None admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:49.618069 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bfb4827-a746-4ea7-898a-a9c8858ee645 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:49.618374 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bfb4827-a746-4ea7-898a-a9c8858ee645 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:49.619801 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bfb4827-a746-4ea7-898a-a9c8858ee645 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:49.628387 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee42255c-1313-4234-82f3-c442448baca7 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:49.636913 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 292/583] 10.222.0.90 () {56 vars in 1148 bytes} [Mon Aug 29 21:20:49 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:49.637520 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bfb4827-a746-4ea7-898a-a9c8858ee645 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:49.667326 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b02800d-d2e9-49c7-9655-a263c8d0a1bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:49.668552 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b02800d-d2e9-49c7-9655-a263c8d0a1bc None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:49.668898 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b02800d-d2e9-49c7-9655-a263c8d0a1bc None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:49.669174 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b02800d-d2e9-49c7-9655-a263c8d0a1bc None admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:49.669843 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b02800d-d2e9-49c7-9655-a263c8d0a1bc None admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:49.670146 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b02800d-d2e9-49c7-9655-a263c8d0a1bc None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:49.671869 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b02800d-d2e9-49c7-9655-a263c8d0a1bc None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:49.689413 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 292/584] 10.222.0.90 () {56 vars in 1109 bytes} [Mon Aug 29 21:20:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:49.689413 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b02800d-d2e9-49c7-9655-a263c8d0a1bc None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:49.737874 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:49.746002 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bfb4827-a746-4ea7-898a-a9c8858ee645 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:49.753540 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:49.783324 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccffc6d5-aff0-4b75-9958-02afd6d9df33 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:49.784675 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccffc6d5-aff0-4b75-9958-02afd6d9df33 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:49.784788 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccffc6d5-aff0-4b75-9958-02afd6d9df33 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:49.785045 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccffc6d5-aff0-4b75-9958-02afd6d9df33 None admin] PATH_INFO: `/v3/users/glance` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:49.793078 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccffc6d5-aff0-4b75-9958-02afd6d9df33 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:49.794547 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccffc6d5-aff0-4b75-9958-02afd6d9df33 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:49.797800 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b02800d-d2e9-49c7-9655-a263c8d0a1bc None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:49.806386 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-ccffc6d5-aff0-4b75-9958-02afd6d9df33 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Aug 29 21:20:49.806386 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:49.806386 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:49.806386 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:49.806386 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:49.806386 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:49.806386 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:49.806386 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:49.806386 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:49.806386 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:49.806386 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:49.806386 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:49.806386 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 29 21:20:49.806386 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 29 21:20:49.806386 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 29 21:20:49.806386 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 29 21:20:49.807343 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:49.807343 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:49.807343 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 29 21:20:49.807343 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 21:20:49.807343 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 29 21:20:49.807343 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 21:20:49.807343 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:49.807343 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:49.807343 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:49.807343 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:49.807343 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:49.807343 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:49.807343 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:49.807343 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:49.807343 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:49.807343 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:49.807343 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:49.807343 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:49.808301 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:49.808301 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:49.808301 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 29 21:20:49.808301 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 29 21:20:49.808301 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 29 21:20:49.808301 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 29 21:20:49.808301 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 29 21:20:49.808301 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 29 21:20:49.808301 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Aug 29 21:20:49.808301 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:20:49.808301 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 293/585] 10.222.0.90 () {56 vars in 1099 bytes} [Mon Aug 29 21:20:49 2022] GET /identity/v3/users/glance => generated 84 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:20:49.808301 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccffc6d5-aff0-4b75-9958-02afd6d9df33 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:49.839653 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbe5c181-15d6-4e44-8843-7cb65aaef687 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:49.840267 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe5c181-15d6-4e44-8843-7cb65aaef687 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:49.840578 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe5c181-15d6-4e44-8843-7cb65aaef687 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:49.840856 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe5c181-15d6-4e44-8843-7cb65aaef687 None admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:49.841518 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbe5c181-15d6-4e44-8843-7cb65aaef687 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:49.841865 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbe5c181-15d6-4e44-8843-7cb65aaef687 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:49.843251 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbe5c181-15d6-4e44-8843-7cb65aaef687 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:49.863151 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 293/586] 10.222.0.90 () {56 vars in 1109 bytes} [Mon Aug 29 21:20:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:49.864297 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbe5c181-15d6-4e44-8843-7cb65aaef687 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:49.928193 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccffc6d5-aff0-4b75-9958-02afd6d9df33 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:49.962063 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5719dbcc-5689-442e-93ec-d70a60e0ba53 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:49.962728 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5719dbcc-5689-442e-93ec-d70a60e0ba53 None admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:49.963007 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5719dbcc-5689-442e-93ec-d70a60e0ba53 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:49.963279 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5719dbcc-5689-442e-93ec-d70a60e0ba53 None admin] PATH_INFO: `/v3/projects/ec03e112e3d547fbab6faf7b485c3fca/users/a35b5b7dea7b421da056f063d2436ffb/roles/a94bdb4c5796428baf63644c062dfcb2` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:49.976258 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbe5c181-15d6-4e44-8843-7cb65aaef687 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:49.978589 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5719dbcc-5689-442e-93ec-d70a60e0ba53 None admin] RBAC: Authorizing `identity:create_grant(project_id=ec03e112e3d547fbab6faf7b485c3fca, user_id=a35b5b7dea7b421da056f063d2436ffb, role_id=a94bdb4c5796428baf63644c062dfcb2)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:49.980038 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5719dbcc-5689-442e-93ec-d70a60e0ba53 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:49.984671 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:49.994837 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 294/587] 10.222.0.90 () {58 vars in 1439 bytes} [Mon Aug 29 21:20:49 2022] PUT /identity/v3/projects/ec03e112e3d547fbab6faf7b485c3fca/users/a35b5b7dea7b421da056f063d2436ffb/roles/a94bdb4c5796428baf63644c062dfcb2 => generated 0 bytes in 187 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:20:49.995606 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5719dbcc-5689-442e-93ec-d70a60e0ba53 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:50.030271 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1c736bf-4f7e-4a1d-8538-12953236c578 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:50.031719 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1c736bf-4f7e-4a1d-8538-12953236c578 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:50.032485 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1c736bf-4f7e-4a1d-8538-12953236c578 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:50.032996 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1c736bf-4f7e-4a1d-8538-12953236c578 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:50.044764 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1c736bf-4f7e-4a1d-8538-12953236c578 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:50.045301 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:50.047624 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1c736bf-4f7e-4a1d-8538-12953236c578 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:50.065501 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-a1c736bf-4f7e-4a1d-8538-12953236c578 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Aug 29 21:20:50.065501 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:50.065501 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:50.065501 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:50.065501 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:50.065501 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:50.065501 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:50.065501 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:50.065501 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:50.065501 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:50.065501 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:50.065501 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:50.065501 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 29 21:20:50.065501 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 29 21:20:50.065501 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 29 21:20:50.065501 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 29 21:20:50.066941 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:50.066941 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:50.066941 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 29 21:20:50.066941 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 21:20:50.066941 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 29 21:20:50.066941 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 21:20:50.066941 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:50.066941 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:50.066941 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:50.066941 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:50.066941 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:50.066941 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:50.066941 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:50.066941 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:50.066941 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:50.066941 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:50.066941 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:50.066941 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:50.068130 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:50.068130 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:50.068130 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 29 21:20:50.068130 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 29 21:20:50.068130 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 29 21:20:50.068130 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 29 21:20:50.068130 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 29 21:20:50.068130 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 29 21:20:50.068130 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Aug 29 21:20:50.068130 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:20:50.068966 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 294/588] 10.222.0.90 () {56 vars in 1099 bytes} [Mon Aug 29 21:20:49 2022] GET /identity/v3/users/cinder => generated 84 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:20:50.070093 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1c736bf-4f7e-4a1d-8538-12953236c578 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:50.133130 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5719dbcc-5689-442e-93ec-d70a60e0ba53 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:50.187504 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e1d1c2a-53ed-40ad-8214-ede6c1813a55 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:50.188333 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e1d1c2a-53ed-40ad-8214-ede6c1813a55 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:50.188713 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e1d1c2a-53ed-40ad-8214-ede6c1813a55 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:50.189090 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e1d1c2a-53ed-40ad-8214-ede6c1813a55 None admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:50.190570 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e1d1c2a-53ed-40ad-8214-ede6c1813a55 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:50.191030 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e1d1c2a-53ed-40ad-8214-ede6c1813a55 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:50.193038 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e1d1c2a-53ed-40ad-8214-ede6c1813a55 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:50.202479 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1c736bf-4f7e-4a1d-8538-12953236c578 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:50.217749 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 295/589] 10.222.0.90 () {56 vars in 1137 bytes} [Mon Aug 29 21:20:49 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:50.219685 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e1d1c2a-53ed-40ad-8214-ede6c1813a55 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:50.240137 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0814d92e-3a26-4d01-bacd-0720bf575499 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:50.241132 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0814d92e-3a26-4d01-bacd-0720bf575499 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:50.241403 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0814d92e-3a26-4d01-bacd-0720bf575499 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:50.241676 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0814d92e-3a26-4d01-bacd-0720bf575499 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:50.250060 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0814d92e-3a26-4d01-bacd-0720bf575499 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:50.251432 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0814d92e-3a26-4d01-bacd-0720bf575499 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:50.269243 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-0814d92e-3a26-4d01-bacd-0720bf575499 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Aug 29 21:20:50.269243 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:50.269243 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:50.269243 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:50.269243 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:50.269243 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:50.269243 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:50.269243 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:50.269243 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:50.269243 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:50.269243 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:50.269243 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:50.269243 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 29 21:20:50.269243 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 29 21:20:50.269243 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 29 21:20:50.269243 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 29 21:20:50.270262 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:50.270262 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:50.270262 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 29 21:20:50.270262 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 21:20:50.270262 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 29 21:20:50.270262 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 21:20:50.270262 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:50.270262 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:50.270262 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:50.270262 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:50.270262 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:50.270262 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:50.270262 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:50.270262 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:50.270262 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:50.270262 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:50.270262 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:50.270262 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:50.271622 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:50.271622 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:50.271622 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 29 21:20:50.271622 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 29 21:20:50.271622 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 29 21:20:50.271622 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 29 21:20:50.271622 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 29 21:20:50.271622 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 29 21:20:50.271622 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Aug 29 21:20:50.271622 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:20:50.271622 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 295/590] 10.222.0.90 () {56 vars in 1102 bytes} [Mon Aug 29 21:20:50 2022] GET /identity/v3/users/neutron => generated 85 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:20:50.271622 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0814d92e-3a26-4d01-bacd-0720bf575499 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:50.305156 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:50.325820 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:50.358236 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e1d1c2a-53ed-40ad-8214-ede6c1813a55 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:50.388256 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0814d92e-3a26-4d01-bacd-0720bf575499 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:50.394338 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20cce65f-7cd9-46b1-8d74-dafce4a20d4a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:50.395056 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20cce65f-7cd9-46b1-8d74-dafce4a20d4a None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:50.395332 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20cce65f-7cd9-46b1-8d74-dafce4a20d4a None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:50.395598 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20cce65f-7cd9-46b1-8d74-dafce4a20d4a None admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:50.396319 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20cce65f-7cd9-46b1-8d74-dafce4a20d4a None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:50.396738 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20cce65f-7cd9-46b1-8d74-dafce4a20d4a None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:50.398195 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20cce65f-7cd9-46b1-8d74-dafce4a20d4a None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:50.420221 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 296/591] 10.222.0.90 () {56 vars in 1137 bytes} [Mon Aug 29 21:20:50 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:50.420221 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20cce65f-7cd9-46b1-8d74-dafce4a20d4a None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:50.431470 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b94ba533-cda7-412f-b227-9eed25bd3b46 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:50.432449 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b94ba533-cda7-412f-b227-9eed25bd3b46 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:50.432789 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b94ba533-cda7-412f-b227-9eed25bd3b46 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:50.433065 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b94ba533-cda7-412f-b227-9eed25bd3b46 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:50.445861 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b94ba533-cda7-412f-b227-9eed25bd3b46 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:50.447390 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b94ba533-cda7-412f-b227-9eed25bd3b46 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:50.463572 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-b94ba533-cda7-412f-b227-9eed25bd3b46 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:20:50.463572 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:50.463572 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:50.463572 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:50.463572 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:50.463572 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:50.463572 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:50.463572 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:50.463572 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:50.463572 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:50.463572 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:50.463572 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:50.463572 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 21:20:50.463572 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 21:20:50.463572 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 21:20:50.463572 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 21:20:50.464770 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:50.464770 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:50.464770 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:50.464770 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:50.464770 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:50.464770 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:50.464770 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:50.464770 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:50.464770 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:50.464770 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:50.464770 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:50.464770 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:50.464770 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:50.464770 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:50.464770 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:50.464770 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:50.464770 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 21:20:50.465694 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 21:20:50.465694 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:20:50.465694 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:20:50.465694 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 296/592] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:20:50 2022] GET /identity/v3/domains/Default => generated 87 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:20:50.466416 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-961773d1-25ac-4f6d-9fd7-bcdc3bed34a7 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:50.466416 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-961773d1-25ac-4f6d-9fd7-bcdc3bed34a7 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:50.466681 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-961773d1-25ac-4f6d-9fd7-bcdc3bed34a7 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:50.470196 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 297/593] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:20:50 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:20:50.471218 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-961773d1-25ac-4f6d-9fd7-bcdc3bed34a7 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:50.533570 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20cce65f-7cd9-46b1-8d74-dafce4a20d4a None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:50.562303 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:50.567352 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:50.574103 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-227e1a32-be71-47b5-9246-9096bb855083 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:50.574899 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-227e1a32-be71-47b5-9246-9096bb855083 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:50.575746 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-227e1a32-be71-47b5-9246-9096bb855083 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:50.576021 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-227e1a32-be71-47b5-9246-9096bb855083 None admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:50.576746 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-227e1a32-be71-47b5-9246-9096bb855083 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:50.577052 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-227e1a32-be71-47b5-9246-9096bb855083 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:50.578495 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-227e1a32-be71-47b5-9246-9096bb855083 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:50.589338 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-961773d1-25ac-4f6d-9fd7-bcdc3bed34a7 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:50.597468 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 297/594] 10.222.0.90 () {56 vars in 1139 bytes} [Mon Aug 29 21:20:50 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:50.602571 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-227e1a32-be71-47b5-9246-9096bb855083 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:50.626451 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edee76cf-d464-4d36-8c6c-6e5fb01f9281 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:50.627212 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edee76cf-d464-4d36-8c6c-6e5fb01f9281 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:50.627615 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edee76cf-d464-4d36-8c6c-6e5fb01f9281 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:50.628027 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edee76cf-d464-4d36-8c6c-6e5fb01f9281 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:50.639531 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edee76cf-d464-4d36-8c6c-6e5fb01f9281 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:50.641202 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edee76cf-d464-4d36-8c6c-6e5fb01f9281 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:50.657154 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-edee76cf-d464-4d36-8c6c-6e5fb01f9281 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:20:50.657154 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:50.657154 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:50.657154 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:50.657154 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:50.657154 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:50.657154 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:50.657154 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:50.657154 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:50.657154 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:50.657154 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:50.657154 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:50.657154 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 21:20:50.657154 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 21:20:50.657154 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 21:20:50.657154 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 21:20:50.658456 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:50.658456 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:50.658456 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:50.658456 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:50.658456 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:50.658456 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:50.658456 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:50.658456 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:50.658456 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:50.658456 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:50.658456 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:50.658456 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:50.658456 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:50.658456 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:50.658456 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:50.658456 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:50.658456 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 21:20:50.660139 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 21:20:50.660139 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:20:50.660139 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:20:50.660139 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 298/595] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:20:50 2022] GET /identity/v3/domains/Default => generated 87 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:20:50.661157 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84029ef7-bb17-464b-8928-c0c70762873a None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:50.661479 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84029ef7-bb17-464b-8928-c0c70762873a None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:50.661767 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84029ef7-bb17-464b-8928-c0c70762873a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:50.681338 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-84029ef7-bb17-464b-8928-c0c70762873a None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:20:50.682976 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-84029ef7-bb17-464b-8928-c0c70762873a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:20:50.715537 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-227e1a32-be71-47b5-9246-9096bb855083 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:50.751925 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8655b27c-a14a-4d89-9615-803a842b2139 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:50.752576 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8655b27c-a14a-4d89-9615-803a842b2139 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:50.753546 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8655b27c-a14a-4d89-9615-803a842b2139 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:50.754043 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8655b27c-a14a-4d89-9615-803a842b2139 None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:50.754798 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8655b27c-a14a-4d89-9615-803a842b2139 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:50.755100 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8655b27c-a14a-4d89-9615-803a842b2139 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:50.761995 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 299/596] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:20:50 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 21:20:50.761995 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84029ef7-bb17-464b-8928-c0c70762873a None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:50.762311 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8655b27c-a14a-4d89-9615-803a842b2139 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:50.786984 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 298/597] 10.222.0.90 () {56 vars in 1109 bytes} [Mon Aug 29 21:20:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:50.786984 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8655b27c-a14a-4d89-9615-803a842b2139 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:50.866612 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:50.873051 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:50.923819 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84029ef7-bb17-464b-8928-c0c70762873a None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:50.929862 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8655b27c-a14a-4d89-9615-803a842b2139 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:50.961678 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18e5a463-0fa4-4739-a0e8-d1ab8f5935ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:50.962667 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e5a463-0fa4-4739-a0e8-d1ab8f5935ca None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:50.963237 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e5a463-0fa4-4739-a0e8-d1ab8f5935ca None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:50.963719 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e5a463-0fa4-4739-a0e8-d1ab8f5935ca None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:50.966372 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0353b55d-8de9-4642-b30b-2f042d760bd5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:50.966996 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0353b55d-8de9-4642-b30b-2f042d760bd5 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:50.967452 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0353b55d-8de9-4642-b30b-2f042d760bd5 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:50.967570 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0353b55d-8de9-4642-b30b-2f042d760bd5 None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:50.968170 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0353b55d-8de9-4642-b30b-2f042d760bd5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:50.968478 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0353b55d-8de9-4642-b30b-2f042d760bd5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:50.969870 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0353b55d-8de9-4642-b30b-2f042d760bd5 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:50.976583 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18e5a463-0fa4-4739-a0e8-d1ab8f5935ca None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:50.978747 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18e5a463-0fa4-4739-a0e8-d1ab8f5935ca None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:50.982771 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 299/598] 10.222.0.90 () {56 vars in 1109 bytes} [Mon Aug 29 21:20:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:50.983687 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0353b55d-8de9-4642-b30b-2f042d760bd5 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:50.996191 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-18e5a463-0fa4-4739-a0e8-d1ab8f5935ca None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:20:50.996191 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:50.996191 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:50.996191 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:50.996191 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:50.996191 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:50.996191 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:50.996191 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:50.996191 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:50.996191 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:50.996191 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:50.996191 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:50.996191 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 21:20:50.996191 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 21:20:50.996191 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 21:20:50.996191 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 21:20:50.997213 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:50.997213 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:50.997213 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:50.997213 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:50.997213 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:50.997213 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:50.997213 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:50.997213 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:50.997213 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:50.997213 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:50.997213 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:50.997213 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:50.997213 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:50.997213 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:50.997213 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:50.997213 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:50.997213 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 21:20:50.998276 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 21:20:50.998276 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:20:50.998276 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:20:50.999190 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 300/599] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:20:50 2022] GET /identity/v3/domains/Default => generated 87 bytes in 243 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:20:51.004734 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18e5a463-0fa4-4739-a0e8-d1ab8f5935ca None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:51.112581 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18e5a463-0fa4-4739-a0e8-d1ab8f5935ca None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:51.115148 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0353b55d-8de9-4642-b30b-2f042d760bd5 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:51.127500 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:51.127776 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:51.153177 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f30aed31-afad-4ee3-8ece-8643bb2a6b32 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:51.153802 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f30aed31-afad-4ee3-8ece-8643bb2a6b32 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:51.154092 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f30aed31-afad-4ee3-8ece-8643bb2a6b32 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:51.154359 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f30aed31-afad-4ee3-8ece-8643bb2a6b32 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:51.155542 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49734d7e-9242-4760-98ac-e67d4f5608d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:51.156490 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49734d7e-9242-4760-98ac-e67d4f5608d2 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:51.157059 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49734d7e-9242-4760-98ac-e67d4f5608d2 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:51.157515 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49734d7e-9242-4760-98ac-e67d4f5608d2 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:51.161693 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f30aed31-afad-4ee3-8ece-8643bb2a6b32 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:51.163150 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f30aed31-afad-4ee3-8ece-8643bb2a6b32 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:51.165528 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49734d7e-9242-4760-98ac-e67d4f5608d2 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:51.167042 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49734d7e-9242-4760-98ac-e67d4f5608d2 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:51.175206 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-f30aed31-afad-4ee3-8ece-8643bb2a6b32 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 21:20:51.175206 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:51.175206 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:51.175206 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:51.175206 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:51.175206 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:51.175206 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:51.175206 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:51.175206 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:51.175206 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:51.175206 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:51.175206 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:51.175206 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 29 21:20:51.175206 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 29 21:20:51.175206 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 29 21:20:51.176121 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 29 21:20:51.176121 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:51.176121 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:51.176121 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:51.176121 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:51.176121 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:51.176121 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:51.176121 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:51.176121 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:51.176121 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:51.176121 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:51.176121 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:51.176121 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:51.176121 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:51.176121 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:51.176121 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:51.176121 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:51.177623 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 29 21:20:51.177623 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 29 21:20:51.177623 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 29 21:20:51.177623 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 29 21:20:51.177623 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 29 21:20:51.177623 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 29 21:20:51.177623 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 21:20:51.177623 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:20:51.177623 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 300/600] 10.222.0.90 () {56 vars in 1111 bytes} [Mon Aug 29 21:20:50 2022] GET /identity/v3/projects/service => generated 88 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:20:51.178566 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-49734d7e-9242-4760-98ac-e67d4f5608d2 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 21:20:51.178566 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:51.178566 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:51.178566 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:51.178566 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:51.178566 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:51.178566 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:51.178566 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:51.178566 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:51.178566 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:51.178566 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:51.178566 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:51.178566 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 29 21:20:51.178566 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 29 21:20:51.178566 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 29 21:20:51.179400 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 29 21:20:51.179400 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:51.179400 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:51.179400 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:51.179400 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:51.179400 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:51.179400 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:51.179400 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:51.179400 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:51.179400 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:51.179400 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:51.179400 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:51.179400 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:51.179400 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:51.179400 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:51.179400 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:51.179400 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:51.181270 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 29 21:20:51.181270 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 29 21:20:51.181270 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 29 21:20:51.181270 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 29 21:20:51.181270 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 29 21:20:51.181270 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 29 21:20:51.181270 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 21:20:51.181270 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:20:51.181270 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 301/601] 10.222.0.90 () {56 vars in 1111 bytes} [Mon Aug 29 21:20:50 2022] GET /identity/v3/projects/service => generated 88 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:20:51.181270 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49734d7e-9242-4760-98ac-e67d4f5608d2 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:51.184897 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f30aed31-afad-4ee3-8ece-8643bb2a6b32 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:51.289244 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49734d7e-9242-4760-98ac-e67d4f5608d2 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:51.294540 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f30aed31-afad-4ee3-8ece-8643bb2a6b32 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:51.324333 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2f55800-844e-42ad-bedd-1e68d4cf131b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:51.325841 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2f55800-844e-42ad-bedd-1e68d4cf131b None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:51.326229 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2f55800-844e-42ad-bedd-1e68d4cf131b None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:51.326608 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2f55800-844e-42ad-bedd-1e68d4cf131b None admin] PATH_INFO: `/v3/roles/service` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:51.334502 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0656090d-a72f-4b3a-9104-6a562d2ce60d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:51.335193 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0656090d-a72f-4b3a-9104-6a562d2ce60d None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:51.335479 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0656090d-a72f-4b3a-9104-6a562d2ce60d None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:51.335754 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0656090d-a72f-4b3a-9104-6a562d2ce60d None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:51.336395 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0656090d-a72f-4b3a-9104-6a562d2ce60d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:51.336699 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0656090d-a72f-4b3a-9104-6a562d2ce60d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:51.338096 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0656090d-a72f-4b3a-9104-6a562d2ce60d None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:51.338322 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2f55800-844e-42ad-bedd-1e68d4cf131b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 21:20:51.343668 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-b2f55800-844e-42ad-bedd-1e68d4cf131b None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 29 21:20:51.343668 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:51.343668 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:51.343668 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:51.343668 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:51.343668 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:51.343668 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:51.343668 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:51.343668 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:51.343668 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:51.343668 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:51.343668 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:51.343668 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 21:20:51.343668 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 21:20:51.343668 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 21:20:51.343668 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 21:20:51.344840 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 21:20:51.344840 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 21:20:51.344840 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 21:20:51.344840 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 21:20:51.344840 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:51.344840 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:51.344840 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:51.344840 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:51.344840 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:51.344840 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:51.344840 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:51.344840 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:51.344840 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:51.344840 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:51.344840 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:51.344840 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:51.345760 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:51.345760 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:51.345760 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:51.345760 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:51.345760 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 21:20:51.345760 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 21:20:51.345760 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 21:20:51.345760 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 21:20:51.345760 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 21:20:51.345760 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 21:20:51.345760 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 29 21:20:51.345760 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:20:51.345760 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 302/602] 10.222.0.90 () {56 vars in 1102 bytes} [Mon Aug 29 21:20:51 2022] GET /identity/v3/roles/service => generated 85 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:20:51.347927 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2f55800-844e-42ad-bedd-1e68d4cf131b None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:51.355868 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 301/603] 10.222.0.90 () {56 vars in 1109 bytes} [Mon Aug 29 21:20:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:51.357579 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0656090d-a72f-4b3a-9104-6a562d2ce60d None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:51.380410 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:51.394322 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:51.454581 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2f55800-844e-42ad-bedd-1e68d4cf131b None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:51.475597 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0656090d-a72f-4b3a-9104-6a562d2ce60d None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:51.494275 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcf699bb-3ab8-40cd-b143-985accfd054b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:51.494963 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcf699bb-3ab8-40cd-b143-985accfd054b None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:51.495263 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcf699bb-3ab8-40cd-b143-985accfd054b None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:51.495536 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcf699bb-3ab8-40cd-b143-985accfd054b None admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:51.496235 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcf699bb-3ab8-40cd-b143-985accfd054b None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:51.496538 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcf699bb-3ab8-40cd-b143-985accfd054b None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:51.498027 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcf699bb-3ab8-40cd-b143-985accfd054b None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:51.515795 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 303/604] 10.222.0.90 () {56 vars in 1148 bytes} [Mon Aug 29 21:20:51 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:51.515795 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcf699bb-3ab8-40cd-b143-985accfd054b None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:51.518162 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b03a85f-f6d1-479e-97ba-cf12e59e6571 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:51.519031 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b03a85f-f6d1-479e-97ba-cf12e59e6571 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:51.519542 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b03a85f-f6d1-479e-97ba-cf12e59e6571 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:51.519968 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b03a85f-f6d1-479e-97ba-cf12e59e6571 None admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:51.520823 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b03a85f-f6d1-479e-97ba-cf12e59e6571 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:51.521133 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b03a85f-f6d1-479e-97ba-cf12e59e6571 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:51.522593 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b03a85f-f6d1-479e-97ba-cf12e59e6571 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:51.537125 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 302/605] 10.222.0.90 () {56 vars in 1148 bytes} [Mon Aug 29 21:20:51 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:51.538041 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b03a85f-f6d1-479e-97ba-cf12e59e6571 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:51.614141 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:51.623908 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcf699bb-3ab8-40cd-b143-985accfd054b None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:51.640907 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:51.652157 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b03a85f-f6d1-479e-97ba-cf12e59e6571 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:51.660841 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25dc7bcc-4cf1-44fa-bf41-27b64dca90a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:51.661461 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25dc7bcc-4cf1-44fa-bf41-27b64dca90a9 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:51.661738 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25dc7bcc-4cf1-44fa-bf41-27b64dca90a9 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:51.662039 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25dc7bcc-4cf1-44fa-bf41-27b64dca90a9 None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:51.662711 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25dc7bcc-4cf1-44fa-bf41-27b64dca90a9 None admin] RBAC: Adding query filter params (name=service) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:51.663014 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25dc7bcc-4cf1-44fa-bf41-27b64dca90a9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:51.664369 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25dc7bcc-4cf1-44fa-bf41-27b64dca90a9 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:51.677604 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 304/606] 10.222.0.90 () {56 vars in 1103 bytes} [Mon Aug 29 21:20:51 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:51.677604 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25dc7bcc-4cf1-44fa-bf41-27b64dca90a9 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:51.692624 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c56ab5e6-641c-4823-a969-188e242d77d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:51.693346 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c56ab5e6-641c-4823-a969-188e242d77d6 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:51.693626 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c56ab5e6-641c-4823-a969-188e242d77d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:51.693947 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c56ab5e6-641c-4823-a969-188e242d77d6 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:51.700671 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c56ab5e6-641c-4823-a969-188e242d77d6 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:51.702170 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c56ab5e6-641c-4823-a969-188e242d77d6 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:51.714282 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-c56ab5e6-641c-4823-a969-188e242d77d6 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 21:20:51.714282 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:51.714282 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:51.714282 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:51.714282 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:51.714282 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:51.714282 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:51.714282 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:51.714282 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:51.714282 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:51.714282 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:51.714282 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:51.714282 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 29 21:20:51.714282 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 29 21:20:51.714282 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 29 21:20:51.720003 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 29 21:20:51.720003 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:51.720003 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:51.720003 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:51.720003 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:51.720003 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:51.720003 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:51.720003 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:51.720003 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:51.720003 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:51.720003 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:51.720003 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:51.720003 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:51.720003 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:51.720003 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:51.720003 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:51.720003 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:51.721201 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 29 21:20:51.721201 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 29 21:20:51.721201 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 29 21:20:51.721201 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 29 21:20:51.721201 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 29 21:20:51.721201 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 29 21:20:51.721201 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 21:20:51.721201 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:20:51.721201 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 303/607] 10.222.0.90 () {56 vars in 1111 bytes} [Mon Aug 29 21:20:51 2022] GET /identity/v3/projects/service => generated 88 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:20:51.721201 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c56ab5e6-641c-4823-a969-188e242d77d6 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:51.786887 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25dc7bcc-4cf1-44fa-bf41-27b64dca90a9 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:51.820476 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1e3164c-139c-4c33-b088-86a07cbc235e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:51.821144 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e3164c-139c-4c33-b088-86a07cbc235e None admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:51.821424 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e3164c-139c-4c33-b088-86a07cbc235e None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:51.821702 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e3164c-139c-4c33-b088-86a07cbc235e None admin] PATH_INFO: `/v3/projects/ec03e112e3d547fbab6faf7b485c3fca/users/0dc226f7ddcb49ecacb51ce57fdfc11b/roles/a94bdb4c5796428baf63644c062dfcb2` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:51.828646 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c56ab5e6-641c-4823-a969-188e242d77d6 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:51.838705 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1e3164c-139c-4c33-b088-86a07cbc235e None admin] RBAC: Authorizing `identity:create_grant(project_id=ec03e112e3d547fbab6faf7b485c3fca, user_id=0dc226f7ddcb49ecacb51ce57fdfc11b, role_id=a94bdb4c5796428baf63644c062dfcb2)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:51.840203 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1e3164c-139c-4c33-b088-86a07cbc235e None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:51.865207 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 305/608] 10.222.0.90 () {58 vars in 1439 bytes} [Mon Aug 29 21:20:51 2022] PUT /identity/v3/projects/ec03e112e3d547fbab6faf7b485c3fca/users/0dc226f7ddcb49ecacb51ce57fdfc11b/roles/a94bdb4c5796428baf63644c062dfcb2 => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:20:51.865207 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1e3164c-139c-4c33-b088-86a07cbc235e None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:51.865207 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:51.889126 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3215e0b9-019d-47dd-9a0d-2655638fec2f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:51.890188 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3215e0b9-019d-47dd-9a0d-2655638fec2f None admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:51.890794 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3215e0b9-019d-47dd-9a0d-2655638fec2f None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:51.891233 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3215e0b9-019d-47dd-9a0d-2655638fec2f None admin] PATH_INFO: `/v3/projects/ec03e112e3d547fbab6faf7b485c3fca/users/c63a4eaf48ad43529f0aa328f2ebfd29/roles/a94bdb4c5796428baf63644c062dfcb2` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:51.914129 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3215e0b9-019d-47dd-9a0d-2655638fec2f None admin] RBAC: Authorizing `identity:create_grant(project_id=ec03e112e3d547fbab6faf7b485c3fca, user_id=c63a4eaf48ad43529f0aa328f2ebfd29, role_id=a94bdb4c5796428baf63644c062dfcb2)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:51.916119 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3215e0b9-019d-47dd-9a0d-2655638fec2f None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:51.939285 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 304/609] 10.222.0.90 () {58 vars in 1439 bytes} [Mon Aug 29 21:20:51 2022] PUT /identity/v3/projects/ec03e112e3d547fbab6faf7b485c3fca/users/c63a4eaf48ad43529f0aa328f2ebfd29/roles/a94bdb4c5796428baf63644c062dfcb2 => generated 0 bytes in 223 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:20:51.950465 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:51.952404 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3215e0b9-019d-47dd-9a0d-2655638fec2f None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:52.026830 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1e3164c-139c-4c33-b088-86a07cbc235e None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:52.081227 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ca5f1fa-f78e-4a51-a6af-3129832cf852 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:52.082092 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca5f1fa-f78e-4a51-a6af-3129832cf852 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:52.082368 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca5f1fa-f78e-4a51-a6af-3129832cf852 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:52.082640 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca5f1fa-f78e-4a51-a6af-3129832cf852 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:52.098549 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ca5f1fa-f78e-4a51-a6af-3129832cf852 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:52.100342 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ca5f1fa-f78e-4a51-a6af-3129832cf852 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:52.101183 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3215e0b9-019d-47dd-9a0d-2655638fec2f None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:52.120367 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-2ca5f1fa-f78e-4a51-a6af-3129832cf852 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:20:52.120367 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:52.120367 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:52.120367 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:52.120367 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:52.120367 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:52.120367 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:52.120367 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:52.120367 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:52.120367 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:52.120367 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:52.120367 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:52.120367 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 21:20:52.120367 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 21:20:52.120367 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 21:20:52.120367 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 21:20:52.121382 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:52.121382 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:52.121382 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:52.121382 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:52.121382 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:52.121382 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:52.121382 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:52.121382 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:52.121382 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:52.121382 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:52.121382 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:52.121382 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:52.121382 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:52.121382 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:52.121382 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:52.121382 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:52.121382 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 21:20:52.122881 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 21:20:52.122881 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:20:52.122881 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:20:52.122881 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 306/610] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:20:51 2022] GET /identity/v3/domains/Default => generated 87 bytes in 263 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:20:52.123227 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ed6627c-e7e0-4d34-a1f3-b6c4c88bef27 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:52.124287 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ed6627c-e7e0-4d34-a1f3-b6c4c88bef27 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:52.124287 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ed6627c-e7e0-4d34-a1f3-b6c4c88bef27 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:52.125341 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 307/611] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:20:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:20:52.130796 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ed6627c-e7e0-4d34-a1f3-b6c4c88bef27 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:52.144951 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-289a794e-190a-40e1-ac47-4953c078f604 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:52.145497 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-289a794e-190a-40e1-ac47-4953c078f604 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:52.146106 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-289a794e-190a-40e1-ac47-4953c078f604 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:52.146381 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-289a794e-190a-40e1-ac47-4953c078f604 None admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:52.147379 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-289a794e-190a-40e1-ac47-4953c078f604 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:52.147687 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-289a794e-190a-40e1-ac47-4953c078f604 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:52.149334 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-289a794e-190a-40e1-ac47-4953c078f604 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:52.165712 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 305/612] 10.222.0.90 () {56 vars in 1148 bytes} [Mon Aug 29 21:20:51 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:52.167662 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb6a005-1b79-4f93-9d6b-02504a0e62bd None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:52.167947 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb6a005-1b79-4f93-9d6b-02504a0e62bd None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:52.168220 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb6a005-1b79-4f93-9d6b-02504a0e62bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:52.192353 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-2fb6a005-1b79-4f93-9d6b-02504a0e62bd None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:20:52.193566 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-2fb6a005-1b79-4f93-9d6b-02504a0e62bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:20:52.218171 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:52.258194 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ed6627c-e7e0-4d34-a1f3-b6c4c88bef27 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:52.269958 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:52.289825 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 306/613] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:20:52 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 21:20:52.289825 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fb6a005-1b79-4f93-9d6b-02504a0e62bd None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:52.318356 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-310c0e09-ced6-4342-ae4e-f95ed5e028c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:52.319374 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-310c0e09-ced6-4342-ae4e-f95ed5e028c3 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:52.319840 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-310c0e09-ced6-4342-ae4e-f95ed5e028c3 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:52.320400 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-310c0e09-ced6-4342-ae4e-f95ed5e028c3 None admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:52.321453 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-310c0e09-ced6-4342-ae4e-f95ed5e028c3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:52.321975 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-310c0e09-ced6-4342-ae4e-f95ed5e028c3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:52.323578 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-310c0e09-ced6-4342-ae4e-f95ed5e028c3 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:52.351769 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 308/614] 10.222.0.90 () {56 vars in 1109 bytes} [Mon Aug 29 21:20:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:52.358265 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-310c0e09-ced6-4342-ae4e-f95ed5e028c3 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:52.403389 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fb6a005-1b79-4f93-9d6b-02504a0e62bd None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:52.466542 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b9d4e38-ac81-47aa-86e9-86728e0e825e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:52.467233 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b9d4e38-ac81-47aa-86e9-86728e0e825e None admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:52.467516 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b9d4e38-ac81-47aa-86e9-86728e0e825e None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:52.467793 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b9d4e38-ac81-47aa-86e9-86728e0e825e None admin] PATH_INFO: `/v3/projects/ec03e112e3d547fbab6faf7b485c3fca/users/0e69b7f0c3574abc9de5fb580d22152a/roles/a94bdb4c5796428baf63644c062dfcb2` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:52.499261 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b9d4e38-ac81-47aa-86e9-86728e0e825e None admin] RBAC: Authorizing `identity:create_grant(project_id=ec03e112e3d547fbab6faf7b485c3fca, user_id=0e69b7f0c3574abc9de5fb580d22152a, role_id=a94bdb4c5796428baf63644c062dfcb2)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:52.500727 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b9d4e38-ac81-47aa-86e9-86728e0e825e None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:52.506474 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-310c0e09-ced6-4342-ae4e-f95ed5e028c3 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:52.520149 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 307/615] 10.222.0.90 () {58 vars in 1439 bytes} [Mon Aug 29 21:20:52 2022] PUT /identity/v3/projects/ec03e112e3d547fbab6faf7b485c3fca/users/0e69b7f0c3574abc9de5fb580d22152a/roles/a94bdb4c5796428baf63644c062dfcb2 => generated 0 bytes in 234 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:20:52.525897 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:52.547613 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65272065-7f85-493f-9a95-79f8fa54e5f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:52.548283 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65272065-7f85-493f-9a95-79f8fa54e5f2 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:52.548621 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65272065-7f85-493f-9a95-79f8fa54e5f2 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:52.548908 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65272065-7f85-493f-9a95-79f8fa54e5f2 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:52.559057 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65272065-7f85-493f-9a95-79f8fa54e5f2 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:52.561395 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65272065-7f85-493f-9a95-79f8fa54e5f2 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:52.576976 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-65272065-7f85-493f-9a95-79f8fa54e5f2 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Aug 29 21:20:52.576976 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:52.576976 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:52.576976 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:52.576976 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:52.576976 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:52.576976 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:52.576976 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:52.576976 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:52.576976 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:52.576976 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:52.576976 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:52.576976 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 29 21:20:52.576976 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 29 21:20:52.576976 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 29 21:20:52.576976 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 29 21:20:52.578079 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:52.578079 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:52.578079 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 29 21:20:52.578079 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 21:20:52.578079 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 29 21:20:52.578079 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 21:20:52.578079 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:52.578079 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:52.578079 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:52.578079 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:52.578079 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:52.578079 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:52.578079 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:52.578079 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:52.578079 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:52.578079 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:52.578079 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:52.578079 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:52.579095 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:52.579095 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:52.579095 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 29 21:20:52.579095 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 29 21:20:52.579095 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 29 21:20:52.579095 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 29 21:20:52.579095 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 29 21:20:52.579095 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 29 21:20:52.579095 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Aug 29 21:20:52.579095 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:20:52.579095 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 309/616] 10.222.0.90 () {56 vars in 1096 bytes} [Mon Aug 29 21:20:52 2022] GET /identity/v3/users/swift => generated 83 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 29 21:20:52.584536 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b9d4e38-ac81-47aa-86e9-86728e0e825e None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:52.653217 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65272065-7f85-493f-9a95-79f8fa54e5f2 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:52.669536 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:52.762699 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b9d4e38-ac81-47aa-86e9-86728e0e825e None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:52.787309 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65272065-7f85-493f-9a95-79f8fa54e5f2 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:52.810955 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8afa7c2e-4af5-4254-82e2-b2b9c49f088f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:52.811586 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8afa7c2e-4af5-4254-82e2-b2b9c49f088f None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:52.811871 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8afa7c2e-4af5-4254-82e2-b2b9c49f088f None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:52.812149 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8afa7c2e-4af5-4254-82e2-b2b9c49f088f None admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:52.812889 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8afa7c2e-4af5-4254-82e2-b2b9c49f088f None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:52.813199 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8afa7c2e-4af5-4254-82e2-b2b9c49f088f None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:52.816059 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8afa7c2e-4af5-4254-82e2-b2b9c49f088f None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:52.834826 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eab16d6-705b-4b98-8b74-14f1a8810de5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:52.836089 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eab16d6-705b-4b98-8b74-14f1a8810de5 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:52.836376 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eab16d6-705b-4b98-8b74-14f1a8810de5 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:52.836650 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eab16d6-705b-4b98-8b74-14f1a8810de5 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:52.839164 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 308/617] 10.222.0.90 () {56 vars in 1135 bytes} [Mon Aug 29 21:20:52 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 255 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:20:52.847783 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8afa7c2e-4af5-4254-82e2-b2b9c49f088f None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:52.861917 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6eab16d6-705b-4b98-8b74-14f1a8810de5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 21:20:52.871925 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-6eab16d6-705b-4b98-8b74-14f1a8810de5 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 29 21:20:52.871925 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:52.871925 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:52.871925 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:52.871925 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:52.871925 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:52.871925 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:52.871925 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:52.871925 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:52.871925 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:52.871925 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:52.871925 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:52.871925 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 21:20:52.871925 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 21:20:52.871925 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 21:20:52.871925 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 21:20:52.872895 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 21:20:52.872895 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 21:20:52.872895 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 21:20:52.872895 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 21:20:52.872895 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:52.872895 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:52.872895 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:52.872895 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:52.872895 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:52.872895 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:52.872895 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:52.872895 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:52.872895 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:52.872895 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:52.872895 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:52.872895 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:52.873748 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:52.873748 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:52.873748 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:52.873748 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:52.873748 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 21:20:52.873748 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 21:20:52.873748 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 21:20:52.873748 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 21:20:52.873748 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 21:20:52.873748 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 21:20:52.873748 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 29 21:20:52.873748 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:20:52.873748 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 310/618] 10.222.0.90 () {56 vars in 1102 bytes} [Mon Aug 29 21:20:52 2022] GET /identity/v3/roles/service => generated 85 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:20:52.882769 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eab16d6-705b-4b98-8b74-14f1a8810de5 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:52.951349 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:53.026324 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:53.036240 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8afa7c2e-4af5-4254-82e2-b2b9c49f088f None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:53.085569 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eab16d6-705b-4b98-8b74-14f1a8810de5 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:53.106344 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ef417c4-55db-45f5-aafd-f16d115e900d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:53.107154 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ef417c4-55db-45f5-aafd-f16d115e900d None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:53.107442 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ef417c4-55db-45f5-aafd-f16d115e900d None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:53.108060 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ef417c4-55db-45f5-aafd-f16d115e900d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:53.127933 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ef417c4-55db-45f5-aafd-f16d115e900d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:53.129757 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ef417c4-55db-45f5-aafd-f16d115e900d None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:53.131798 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-118f388c-9312-4475-86a3-bcc30716ce72 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:53.132519 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-118f388c-9312-4475-86a3-bcc30716ce72 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:53.132809 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-118f388c-9312-4475-86a3-bcc30716ce72 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:53.133090 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-118f388c-9312-4475-86a3-bcc30716ce72 None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:53.133772 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-118f388c-9312-4475-86a3-bcc30716ce72 None admin] RBAC: Adding query filter params (name=service) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:53.135617 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-118f388c-9312-4475-86a3-bcc30716ce72 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:53.137021 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-118f388c-9312-4475-86a3-bcc30716ce72 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:53.146108 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 311/619] 10.222.0.90 () {56 vars in 1103 bytes} [Mon Aug 29 21:20:52 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 264 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:53.152115 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-118f388c-9312-4475-86a3-bcc30716ce72 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:53.156114 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-0ef417c4-55db-45f5-aafd-f16d115e900d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:20:53.156114 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:53.156114 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:53.156114 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:53.156114 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:53.156114 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:53.156114 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:53.156114 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:53.156114 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:53.156114 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:53.156114 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:53.156114 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:53.156114 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 21:20:53.156114 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 21:20:53.156114 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 21:20:53.156114 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 21:20:53.157121 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:53.157121 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:53.157121 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:53.157121 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:53.157121 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:53.157121 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:53.157121 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:53.157121 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:53.157121 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:53.157121 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:53.157121 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:53.157121 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:53.157121 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:53.157121 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:53.157121 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:53.157121 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:53.157121 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 21:20:53.158091 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 21:20:53.158091 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:20:53.158091 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:20:53.158091 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 309/620] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:20:52 2022] GET /identity/v3/domains/Default => generated 87 bytes in 310 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:20:53.168614 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ef417c4-55db-45f5-aafd-f16d115e900d None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:53.290708 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-118f388c-9312-4475-86a3-bcc30716ce72 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:53.292318 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:53.327213 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d020858-a5d3-4ce6-890f-c617f24902ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:53.327858 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d020858-a5d3-4ce6-890f-c617f24902ba None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:53.328142 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d020858-a5d3-4ce6-890f-c617f24902ba None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:53.328419 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d020858-a5d3-4ce6-890f-c617f24902ba None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:53.340551 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d020858-a5d3-4ce6-890f-c617f24902ba None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:53.342066 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d020858-a5d3-4ce6-890f-c617f24902ba None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:53.359220 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-1d020858-a5d3-4ce6-890f-c617f24902ba None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:20:53.359220 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:53.359220 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:53.359220 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:53.359220 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:53.359220 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:53.359220 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:53.359220 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:53.359220 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:53.359220 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:53.359220 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:53.359220 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:53.359220 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 21:20:53.359220 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 21:20:53.359220 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 21:20:53.359220 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 21:20:53.360253 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:53.360253 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:53.360253 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:53.360253 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:53.360253 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:53.360253 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:53.360253 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:53.360253 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:53.360253 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:53.360253 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:53.360253 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:53.360253 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:53.360253 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:53.360253 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:53.360253 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:53.360253 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:53.360253 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 21:20:53.361158 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 21:20:53.361158 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:20:53.361158 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:20:53.361158 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 312/621] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:20:53 2022] GET /identity/v3/domains/Default => generated 87 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 29 21:20:53.369276 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d020858-a5d3-4ce6-890f-c617f24902ba None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:53.415760 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ef417c4-55db-45f5-aafd-f16d115e900d None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:53.466971 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14b0f37a-9636-447f-8ec3-238668531a36 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:53.467592 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b0f37a-9636-447f-8ec3-238668531a36 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:53.467921 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b0f37a-9636-447f-8ec3-238668531a36 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:53.468206 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b0f37a-9636-447f-8ec3-238668531a36 None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:53.468856 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14b0f37a-9636-447f-8ec3-238668531a36 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:53.469162 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14b0f37a-9636-447f-8ec3-238668531a36 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:53.474751 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14b0f37a-9636-447f-8ec3-238668531a36 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:53.499518 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 310/622] 10.222.0.90 () {56 vars in 1109 bytes} [Mon Aug 29 21:20:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 331 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:53.503032 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14b0f37a-9636-447f-8ec3-238668531a36 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:53.523130 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:53.567168 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d020858-a5d3-4ce6-890f-c617f24902ba None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:53.616639 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c692f62b-391a-4059-b41f-c74a0a985f02 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:53.617261 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c692f62b-391a-4059-b41f-c74a0a985f02 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:53.617589 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c692f62b-391a-4059-b41f-c74a0a985f02 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:53.617894 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c692f62b-391a-4059-b41f-c74a0a985f02 None admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:53.618556 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c692f62b-391a-4059-b41f-c74a0a985f02 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:53.618863 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c692f62b-391a-4059-b41f-c74a0a985f02 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:53.620340 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c692f62b-391a-4059-b41f-c74a0a985f02 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:53.633590 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 313/623] 10.222.0.90 () {56 vars in 1109 bytes} [Mon Aug 29 21:20:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 264 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:53.642907 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c692f62b-391a-4059-b41f-c74a0a985f02 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:53.666417 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:53.681280 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14b0f37a-9636-447f-8ec3-238668531a36 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:53.719591 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-133f1b22-df43-4ad1-ad84-5fb7f9122707 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:53.720230 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-133f1b22-df43-4ad1-ad84-5fb7f9122707 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:53.720614 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-133f1b22-df43-4ad1-ad84-5fb7f9122707 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:53.720891 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-133f1b22-df43-4ad1-ad84-5fb7f9122707 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:53.728945 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-133f1b22-df43-4ad1-ad84-5fb7f9122707 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:53.730487 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-133f1b22-df43-4ad1-ad84-5fb7f9122707 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:53.743743 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-133f1b22-df43-4ad1-ad84-5fb7f9122707 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 21:20:53.743743 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:53.743743 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:53.743743 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:53.743743 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:53.743743 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:53.743743 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:53.743743 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:53.743743 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:53.743743 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:53.743743 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:53.743743 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:53.743743 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 29 21:20:53.743743 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 29 21:20:53.743743 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 29 21:20:53.744742 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 29 21:20:53.744742 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:53.744742 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:53.744742 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:53.744742 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:53.744742 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:53.744742 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:53.744742 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:53.744742 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:53.744742 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:53.744742 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:53.744742 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:53.744742 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:53.744742 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:53.744742 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:53.744742 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:53.744742 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:53.745658 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 29 21:20:53.745658 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 29 21:20:53.745658 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 29 21:20:53.745658 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 29 21:20:53.745658 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 29 21:20:53.745658 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 29 21:20:53.745658 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 21:20:53.745658 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:20:53.749752 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 311/624] 10.222.0.90 () {56 vars in 1111 bytes} [Mon Aug 29 21:20:53 2022] GET /identity/v3/projects/service => generated 88 bytes in 243 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:20:53.751065 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-133f1b22-df43-4ad1-ad84-5fb7f9122707 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:53.834291 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:53.873299 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-133f1b22-df43-4ad1-ad84-5fb7f9122707 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:53.912828 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8c76f7d-2d71-4cb9-90c9-de84afd122f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:53.913493 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8c76f7d-2d71-4cb9-90c9-de84afd122f9 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:53.913770 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8c76f7d-2d71-4cb9-90c9-de84afd122f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:53.914069 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8c76f7d-2d71-4cb9-90c9-de84afd122f9 None admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:53.914817 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8c76f7d-2d71-4cb9-90c9-de84afd122f9 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:53.915121 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8c76f7d-2d71-4cb9-90c9-de84afd122f9 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:53.916557 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8c76f7d-2d71-4cb9-90c9-de84afd122f9 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:53.923788 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c692f62b-391a-4059-b41f-c74a0a985f02 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:53.933928 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 312/625] 10.222.0.90 () {56 vars in 1148 bytes} [Mon Aug 29 21:20:53 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:20:53.935879 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8c76f7d-2d71-4cb9-90c9-de84afd122f9 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:54.019185 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45e015c7-7265-4caf-a610-9eeb439cde9a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:54.019965 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45e015c7-7265-4caf-a610-9eeb439cde9a None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:54.020251 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45e015c7-7265-4caf-a610-9eeb439cde9a None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:54.020527 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45e015c7-7265-4caf-a610-9eeb439cde9a None admin] PATH_INFO: `/v3/users/nova` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:54.037809 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45e015c7-7265-4caf-a610-9eeb439cde9a None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:54.039279 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45e015c7-7265-4caf-a610-9eeb439cde9a None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:54.042665 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8c76f7d-2d71-4cb9-90c9-de84afd122f9 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:54.061634 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-45e015c7-7265-4caf-a610-9eeb439cde9a None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 29 21:20:54.061634 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:54.061634 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:54.061634 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:54.061634 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:54.061634 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:54.061634 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:54.061634 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:54.061634 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:54.061634 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:54.061634 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:54.061634 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:54.061634 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 29 21:20:54.061634 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 29 21:20:54.061634 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 29 21:20:54.061634 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 29 21:20:54.062698 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:54.062698 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:54.062698 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 29 21:20:54.062698 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 21:20:54.062698 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 29 21:20:54.062698 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 21:20:54.062698 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:54.062698 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:54.062698 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:54.062698 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:54.062698 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:54.062698 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:54.062698 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:54.062698 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:54.062698 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:54.062698 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:54.062698 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:54.062698 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:54.063708 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:54.063708 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:54.063708 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 29 21:20:54.063708 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 29 21:20:54.063708 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 29 21:20:54.063708 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 29 21:20:54.063708 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 29 21:20:54.063708 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 29 21:20:54.063708 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 29 21:20:54.063708 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:20:54.074887 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 314/626] 10.222.0.90 () {56 vars in 1093 bytes} [Mon Aug 29 21:20:53 2022] GET /identity/v3/users/nova => generated 82 bytes in 433 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:20:54.075018 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:54.082762 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45e015c7-7265-4caf-a610-9eeb439cde9a None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:54.083627 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-503f74a9-6cee-4cf0-be72-7f88f463f33a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:54.084446 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-503f74a9-6cee-4cf0-be72-7f88f463f33a None admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:54.084772 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-503f74a9-6cee-4cf0-be72-7f88f463f33a None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:54.085144 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-503f74a9-6cee-4cf0-be72-7f88f463f33a None admin] PATH_INFO: `/v3/projects/ec03e112e3d547fbab6faf7b485c3fca/users/139bbbde6d884b42b5440727926e2c58/roles/a94bdb4c5796428baf63644c062dfcb2` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:54.102259 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-503f74a9-6cee-4cf0-be72-7f88f463f33a None admin] RBAC: Authorizing `identity:create_grant(project_id=ec03e112e3d547fbab6faf7b485c3fca, user_id=139bbbde6d884b42b5440727926e2c58, role_id=a94bdb4c5796428baf63644c062dfcb2)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:54.103907 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-503f74a9-6cee-4cf0-be72-7f88f463f33a None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:54.121880 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 313/627] 10.222.0.90 () {58 vars in 1439 bytes} [Mon Aug 29 21:20:53 2022] PUT /identity/v3/projects/ec03e112e3d547fbab6faf7b485c3fca/users/139bbbde6d884b42b5440727926e2c58/roles/a94bdb4c5796428baf63644c062dfcb2 => generated 0 bytes in 184 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 21:20:54.123388 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5132a00b-653e-41db-9935-b50d94ee8054 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:54.123682 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5132a00b-653e-41db-9935-b50d94ee8054 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:54.123957 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5132a00b-653e-41db-9935-b50d94ee8054 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:54.125722 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 314/628] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:20:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:20:54.132708 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e2924b-7032-4693-928a-0e2cdc2a5a13 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:54.132991 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e2924b-7032-4693-928a-0e2cdc2a5a13 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:54.133272 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e2924b-7032-4693-928a-0e2cdc2a5a13 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:54.177729 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-66e2924b-7032-4693-928a-0e2cdc2a5a13 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:20:54.180005 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-66e2924b-7032-4693-928a-0e2cdc2a5a13 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:20:54.227688 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:54.290635 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45e015c7-7265-4caf-a610-9eeb439cde9a None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:54.322493 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 315/629] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:20:54 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 190 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 21:20:54.323311 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-960dfd91-25f7-41dd-b425-f34f3b77b7fc None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:54.323600 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-960dfd91-25f7-41dd-b425-f34f3b77b7fc None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:54.323878 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-960dfd91-25f7-41dd-b425-f34f3b77b7fc None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:54.325118 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 316/630] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:20:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:20:54.332834 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-585704e1-31b0-4cf1-afe9-bea8a0858964 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:54.333121 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-585704e1-31b0-4cf1-afe9-bea8a0858964 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:54.333443 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-585704e1-31b0-4cf1-afe9-bea8a0858964 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:54.348891 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22116576-35e7-433d-ad60-cd9e9d1067d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:54.349904 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22116576-35e7-433d-ad60-cd9e9d1067d4 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:54.350695 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22116576-35e7-433d-ad60-cd9e9d1067d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:54.351176 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22116576-35e7-433d-ad60-cd9e9d1067d4 None admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:54.352137 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22116576-35e7-433d-ad60-cd9e9d1067d4 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:54.352654 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22116576-35e7-433d-ad60-cd9e9d1067d4 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:54.354470 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22116576-35e7-433d-ad60-cd9e9d1067d4 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:54.354685 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-585704e1-31b0-4cf1-afe9-bea8a0858964 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:20:54.355897 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-585704e1-31b0-4cf1-afe9-bea8a0858964 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:20:54.383306 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 315/631] 10.222.0.90 () {56 vars in 1133 bytes} [Mon Aug 29 21:20:54 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 297 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:54.390555 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22116576-35e7-433d-ad60-cd9e9d1067d4 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:54.477116 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 317/632] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:20:54 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 146 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 29 21:20:54.537943 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22116576-35e7-433d-ad60-cd9e9d1067d4 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:54.576377 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:54.588234 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a064863-98a1-40f1-8f53-b2178e5a8a0c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:54.588860 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a064863-98a1-40f1-8f53-b2178e5a8a0c None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:54.589433 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a064863-98a1-40f1-8f53-b2178e5a8a0c None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:54.589716 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a064863-98a1-40f1-8f53-b2178e5a8a0c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:54.656600 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-585704e1-31b0-4cf1-afe9-bea8a0858964 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:54.707245 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a064863-98a1-40f1-8f53-b2178e5a8a0c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:54.708714 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a064863-98a1-40f1-8f53-b2178e5a8a0c None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:54.720191 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:54.801730 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-585704e1-31b0-4cf1-afe9-bea8a0858964 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:54.854266 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 316/633] 10.222.0.90 () {58 vars in 1282 bytes} [Mon Aug 29 21:20:54 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 460 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Aug 29 21:20:54.854266 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a064863-98a1-40f1-8f53-b2178e5a8a0c None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:54.861634 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42c2f000-402f-42d3-a2cc-3416ff3eebe4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:54.863748 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42c2f000-402f-42d3-a2cc-3416ff3eebe4 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:54.864210 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42c2f000-402f-42d3-a2cc-3416ff3eebe4 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:54.864644 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42c2f000-402f-42d3-a2cc-3416ff3eebe4 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:54.880023 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42c2f000-402f-42d3-a2cc-3416ff3eebe4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 21:20:54.885306 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:54.889377 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-42c2f000-402f-42d3-a2cc-3416ff3eebe4 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 29 21:20:54.889377 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:54.889377 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:54.889377 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:54.889377 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:54.889377 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:54.889377 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:54.889377 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:54.889377 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:54.889377 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:54.889377 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:54.889377 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:54.889377 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 21:20:54.889377 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 21:20:54.889377 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 21:20:54.889377 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 21:20:54.890630 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 21:20:54.890630 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 21:20:54.890630 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 21:20:54.890630 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 21:20:54.890630 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:54.890630 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:54.890630 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:54.890630 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:54.890630 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:54.890630 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:54.890630 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:54.890630 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:54.890630 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:54.890630 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:54.890630 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:54.890630 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:54.891781 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:54.891781 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:54.891781 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:54.891781 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:54.891781 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 21:20:54.891781 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 21:20:54.891781 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 21:20:54.891781 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 21:20:54.891781 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 21:20:54.891781 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 21:20:54.891781 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 29 21:20:54.891781 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:20:54.895636 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 318/634] 10.222.0.90 () {56 vars in 1102 bytes} [Mon Aug 29 21:20:54 2022] GET /identity/v3/roles/service => generated 85 bytes in 240 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 29 21:20:54.896695 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42c2f000-402f-42d3-a2cc-3416ff3eebe4 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:54.973934 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a064863-98a1-40f1-8f53-b2178e5a8a0c None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:55.010372 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2890f62c-fd9d-4b67-99d2-9260ae756931 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:55.011121 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2890f62c-fd9d-4b67-99d2-9260ae756931 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:55.011485 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2890f62c-fd9d-4b67-99d2-9260ae756931 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:55.011841 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2890f62c-fd9d-4b67-99d2-9260ae756931 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:55.022108 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2890f62c-fd9d-4b67-99d2-9260ae756931 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 21:20:55.024651 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:55.026895 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-2890f62c-fd9d-4b67-99d2-9260ae756931 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 29 21:20:55.026895 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:55.026895 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:55.026895 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:55.026895 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:55.026895 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:55.026895 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:55.026895 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:55.026895 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:55.026895 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:55.026895 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:55.026895 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:55.026895 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 21:20:55.026895 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 21:20:55.026895 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 21:20:55.026895 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 21:20:55.027879 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 21:20:55.027879 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 21:20:55.027879 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 21:20:55.027879 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 21:20:55.027879 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:55.027879 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:55.027879 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:55.027879 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:55.027879 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:55.027879 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:55.027879 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:55.027879 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:55.027879 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:55.027879 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:55.027879 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:55.027879 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:55.028749 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:55.028749 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:55.028749 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:55.028749 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:55.028749 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 21:20:55.028749 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 21:20:55.028749 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 21:20:55.028749 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 21:20:55.028749 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 21:20:55.028749 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 21:20:55.028749 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 29 21:20:55.028749 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:20:55.028749 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 317/635] 10.222.0.90 () {56 vars in 1102 bytes} [Mon Aug 29 21:20:54 2022] GET /identity/v3/roles/service => generated 85 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:20:55.028749 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2890f62c-fd9d-4b67-99d2-9260ae756931 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:55.033301 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42c2f000-402f-42d3-a2cc-3416ff3eebe4 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:55.074357 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49b856d4-81e6-4cef-a651-b610fbefc57c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:55.075320 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b856d4-81e6-4cef-a651-b610fbefc57c None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:55.075607 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b856d4-81e6-4cef-a651-b610fbefc57c None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:55.076057 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b856d4-81e6-4cef-a651-b610fbefc57c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:55.087012 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49b856d4-81e6-4cef-a651-b610fbefc57c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:55.088472 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49b856d4-81e6-4cef-a651-b610fbefc57c None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:55.103761 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-49b856d4-81e6-4cef-a651-b610fbefc57c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:20:55.103761 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:55.103761 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:55.103761 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:55.103761 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:55.103761 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:55.103761 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:55.103761 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:55.103761 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:55.103761 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:55.103761 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:55.103761 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:55.103761 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 21:20:55.103761 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 21:20:55.103761 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 21:20:55.103761 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 21:20:55.104816 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:55.104816 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:55.104816 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:55.104816 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:55.104816 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:55.104816 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:55.104816 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:55.104816 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:55.104816 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:55.104816 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:55.104816 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:55.104816 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:55.104816 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:55.104816 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:55.104816 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:55.104816 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:55.104816 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 21:20:55.106294 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 21:20:55.106294 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:20:55.106294 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:20:55.106294 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 319/636] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:20:54 2022] GET /identity/v3/domains/Default => generated 87 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:20:55.106294 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca651ad4-c6c6-43d3-bad0-53600b86c3ff None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:55.106294 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca651ad4-c6c6-43d3-bad0-53600b86c3ff None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:55.106735 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca651ad4-c6c6-43d3-bad0-53600b86c3ff None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:55.109906 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 320/637] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:20:55 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:20:55.109906 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca651ad4-c6c6-43d3-bad0-53600b86c3ff None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:55.132065 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:55.147424 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2890f62c-fd9d-4b67-99d2-9260ae756931 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:55.184244 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbab31af-efc3-4f98-a713-17e9e9da05ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:55.184869 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbab31af-efc3-4f98-a713-17e9e9da05ef None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:55.185213 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbab31af-efc3-4f98-a713-17e9e9da05ef None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:55.185391 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbab31af-efc3-4f98-a713-17e9e9da05ef None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:55.186080 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbab31af-efc3-4f98-a713-17e9e9da05ef None admin] RBAC: Adding query filter params (name=service) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:55.186380 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbab31af-efc3-4f98-a713-17e9e9da05ef None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:55.188384 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbab31af-efc3-4f98-a713-17e9e9da05ef None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:55.201066 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 318/638] 10.222.0.90 () {56 vars in 1103 bytes} [Mon Aug 29 21:20:55 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:55.201066 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbab31af-efc3-4f98-a713-17e9e9da05ef None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:55.227869 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca651ad4-c6c6-43d3-bad0-53600b86c3ff None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:55.265819 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41e91671-9818-4ecb-99a3-dfd2135e6c15 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:55.266434 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41e91671-9818-4ecb-99a3-dfd2135e6c15 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:55.266713 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41e91671-9818-4ecb-99a3-dfd2135e6c15 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:55.267056 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41e91671-9818-4ecb-99a3-dfd2135e6c15 None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:55.267635 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41e91671-9818-4ecb-99a3-dfd2135e6c15 None admin] RBAC: Adding query filter params (name=service) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:55.267937 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41e91671-9818-4ecb-99a3-dfd2135e6c15 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:55.270044 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41e91671-9818-4ecb-99a3-dfd2135e6c15 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:55.278715 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 321/639] 10.222.0.90 () {56 vars in 1103 bytes} [Mon Aug 29 21:20:55 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:55.279721 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f2948c8-ab39-43bc-b6e6-3fa3f46f05f2 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:55.280005 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f2948c8-ab39-43bc-b6e6-3fa3f46f05f2 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:55.280277 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f2948c8-ab39-43bc-b6e6-3fa3f46f05f2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:55.299518 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-9f2948c8-ab39-43bc-b6e6-3fa3f46f05f2 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:20:55.301743 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-9f2948c8-ab39-43bc-b6e6-3fa3f46f05f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:20:55.314284 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:55.317094 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbab31af-efc3-4f98-a713-17e9e9da05ef None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:55.353798 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc02c6b7-ce0a-46bc-b9e1-b8b2bbde0566 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:55.354410 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc02c6b7-ce0a-46bc-b9e1-b8b2bbde0566 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:55.354689 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc02c6b7-ce0a-46bc-b9e1-b8b2bbde0566 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:55.354961 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc02c6b7-ce0a-46bc-b9e1-b8b2bbde0566 None admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:55.355602 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc02c6b7-ce0a-46bc-b9e1-b8b2bbde0566 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:55.355900 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc02c6b7-ce0a-46bc-b9e1-b8b2bbde0566 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:55.357269 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc02c6b7-ce0a-46bc-b9e1-b8b2bbde0566 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:55.370361 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 319/640] 10.222.0.90 () {56 vars in 1109 bytes} [Mon Aug 29 21:20:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:55.371238 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc02c6b7-ce0a-46bc-b9e1-b8b2bbde0566 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:55.386296 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:55.389393 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 322/641] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:20:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 21:20:55.390210 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f2948c8-ab39-43bc-b6e6-3fa3f46f05f2 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:55.500885 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc02c6b7-ce0a-46bc-b9e1-b8b2bbde0566 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:55.518922 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f2948c8-ab39-43bc-b6e6-3fa3f46f05f2 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:55.537079 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b52acba9-93ec-4486-8eae-c802406a9797 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:55.537740 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b52acba9-93ec-4486-8eae-c802406a9797 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:55.538169 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b52acba9-93ec-4486-8eae-c802406a9797 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:55.538335 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b52acba9-93ec-4486-8eae-c802406a9797 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:55.549295 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b52acba9-93ec-4486-8eae-c802406a9797 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:55.551503 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b52acba9-93ec-4486-8eae-c802406a9797 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:55.557119 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-183862ce-6c88-44a6-acc3-e03077552be6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:55.558060 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-183862ce-6c88-44a6-acc3-e03077552be6 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:55.558535 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-183862ce-6c88-44a6-acc3-e03077552be6 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:55.558984 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-183862ce-6c88-44a6-acc3-e03077552be6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:55.568015 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-b52acba9-93ec-4486-8eae-c802406a9797 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:20:55.568015 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:55.568015 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:55.568015 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:55.568015 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:55.568015 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:55.568015 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:55.568015 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:55.568015 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:55.568015 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:55.568015 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:55.568015 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:55.568015 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 21:20:55.568015 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 21:20:55.568015 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 21:20:55.568015 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 21:20:55.568984 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:55.568984 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:55.568984 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:55.568984 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:55.568984 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:55.568984 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:55.568984 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:55.568984 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:55.568984 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:55.568984 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:55.568984 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:55.568984 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:55.568984 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:55.568984 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:55.568984 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:55.568984 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:55.568984 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 21:20:55.569931 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 21:20:55.569931 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:20:55.569931 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:20:55.573412 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-183862ce-6c88-44a6-acc3-e03077552be6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:55.573990 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 320/642] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:20:55 2022] GET /identity/v3/domains/Default => generated 87 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:20:55.574829 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b52acba9-93ec-4486-8eae-c802406a9797 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:55.575645 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-183862ce-6c88-44a6-acc3-e03077552be6 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:55.578505 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:55.593156 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-183862ce-6c88-44a6-acc3-e03077552be6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:20:55.593156 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:55.593156 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:55.593156 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:55.593156 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:55.593156 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:55.593156 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:55.593156 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:55.593156 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:55.593156 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:55.593156 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:55.593156 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:55.593156 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 21:20:55.593156 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 21:20:55.593156 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 21:20:55.593156 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 21:20:55.594245 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:55.594245 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:55.594245 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:55.594245 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:55.594245 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:55.594245 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:55.594245 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:55.594245 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:55.594245 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:55.594245 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:55.594245 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:55.594245 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:55.594245 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:55.594245 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:55.594245 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:55.594245 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:55.594245 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 21:20:55.595163 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 21:20:55.595163 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:20:55.595163 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:20:55.602450 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 323/643] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:20:55 2022] GET /identity/v3/domains/Default => generated 87 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:20:55.603549 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-183862ce-6c88-44a6-acc3-e03077552be6 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:55.644235 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:55.689042 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b52acba9-93ec-4486-8eae-c802406a9797 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:55.717970 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-183862ce-6c88-44a6-acc3-e03077552be6 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:55.728416 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bc47fcc-7470-44d0-ab18-8f22397aa4b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:55.729381 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bc47fcc-7470-44d0-ab18-8f22397aa4b3 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:55.729847 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bc47fcc-7470-44d0-ab18-8f22397aa4b3 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:55.730134 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bc47fcc-7470-44d0-ab18-8f22397aa4b3 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:55.737776 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bc47fcc-7470-44d0-ab18-8f22397aa4b3 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:55.739256 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bc47fcc-7470-44d0-ab18-8f22397aa4b3 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:55.751190 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-2bc47fcc-7470-44d0-ab18-8f22397aa4b3 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 21:20:55.751190 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:55.751190 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:55.751190 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:55.751190 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:55.751190 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:55.751190 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:55.751190 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:55.751190 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:55.751190 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:55.751190 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:55.751190 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:55.751190 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 29 21:20:55.751190 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 29 21:20:55.751190 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 29 21:20:55.752277 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 29 21:20:55.752277 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:55.752277 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:55.752277 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:55.752277 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:55.752277 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:55.752277 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:55.752277 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:55.752277 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:55.752277 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:55.752277 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:55.752277 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:55.752277 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:55.752277 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:55.752277 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:55.752277 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:55.752277 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:55.754141 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 29 21:20:55.754141 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 29 21:20:55.754141 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 29 21:20:55.754141 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 29 21:20:55.754141 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 29 21:20:55.754141 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 29 21:20:55.754141 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 21:20:55.754141 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:20:55.754141 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 321/644] 10.222.0.90 () {56 vars in 1111 bytes} [Mon Aug 29 21:20:55 2022] GET /identity/v3/projects/service => generated 88 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:20:55.754141 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bc47fcc-7470-44d0-ab18-8f22397aa4b3 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:55.760316 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baea9bb5-2f81-479d-a002-4ac1462e53a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:55.761157 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baea9bb5-2f81-479d-a002-4ac1462e53a1 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:55.761448 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baea9bb5-2f81-479d-a002-4ac1462e53a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:55.761726 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baea9bb5-2f81-479d-a002-4ac1462e53a1 None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:55.762411 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baea9bb5-2f81-479d-a002-4ac1462e53a1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:55.762718 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baea9bb5-2f81-479d-a002-4ac1462e53a1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:55.764088 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baea9bb5-2f81-479d-a002-4ac1462e53a1 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:55.782230 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 324/645] 10.222.0.90 () {56 vars in 1109 bytes} [Mon Aug 29 21:20:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:55.782850 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baea9bb5-2f81-479d-a002-4ac1462e53a1 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:55.847553 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:55.873722 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bc47fcc-7470-44d0-ab18-8f22397aa4b3 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:55.894194 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baea9bb5-2f81-479d-a002-4ac1462e53a1 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:55.897720 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:55.912122 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6f8e4ec-0dc7-464f-9738-758fcc9ec61d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:55.913058 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f8e4ec-0dc7-464f-9738-758fcc9ec61d None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:55.913566 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f8e4ec-0dc7-464f-9738-758fcc9ec61d None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:55.914101 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f8e4ec-0dc7-464f-9738-758fcc9ec61d None admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:55.915408 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6f8e4ec-0dc7-464f-9738-758fcc9ec61d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:55.915895 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6f8e4ec-0dc7-464f-9738-758fcc9ec61d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:55.917719 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6f8e4ec-0dc7-464f-9738-758fcc9ec61d None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:55.932108 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 322/646] 10.222.0.90 () {56 vars in 1109 bytes} [Mon Aug 29 21:20:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:55.932284 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf472041-9bb5-463b-9413-d4d62dd421fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:55.932579 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf472041-9bb5-463b-9413-d4d62dd421fe None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:55.932860 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf472041-9bb5-463b-9413-d4d62dd421fe None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:55.933133 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf472041-9bb5-463b-9413-d4d62dd421fe None admin] PATH_INFO: `/v3/roles/service` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:55.937134 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6f8e4ec-0dc7-464f-9738-758fcc9ec61d None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:55.944154 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf472041-9bb5-463b-9413-d4d62dd421fe None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 21:20:55.948998 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-bf472041-9bb5-463b-9413-d4d62dd421fe None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 29 21:20:55.948998 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:55.948998 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:55.948998 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:55.948998 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:55.948998 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:55.948998 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:55.948998 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:55.948998 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:55.948998 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:55.948998 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:55.948998 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:55.948998 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 21:20:55.948998 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 21:20:55.948998 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 21:20:55.948998 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 21:20:55.949983 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 21:20:55.949983 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 21:20:55.949983 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 21:20:55.949983 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 21:20:55.949983 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:55.949983 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:55.949983 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:55.949983 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:55.949983 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:55.949983 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:55.949983 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:55.949983 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:55.949983 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:55.949983 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:55.949983 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:55.949983 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:55.950875 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:55.950875 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:55.950875 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:55.950875 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:55.950875 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 21:20:55.950875 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 21:20:55.950875 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 21:20:55.950875 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 21:20:55.950875 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 21:20:55.950875 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 21:20:55.950875 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 29 21:20:55.950875 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:20:55.950875 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 325/647] 10.222.0.90 () {56 vars in 1102 bytes} [Mon Aug 29 21:20:55 2022] GET /identity/v3/roles/service => generated 85 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:20:55.951577 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf472041-9bb5-463b-9413-d4d62dd421fe None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:56.047785 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6f8e4ec-0dc7-464f-9738-758fcc9ec61d None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:56.067433 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf472041-9bb5-463b-9413-d4d62dd421fe None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:56.085243 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:56.085957 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9495d72-44b3-4811-8b94-0c3a497044bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:56.086674 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9495d72-44b3-4811-8b94-0c3a497044bc None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:56.086895 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9495d72-44b3-4811-8b94-0c3a497044bc None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:56.087167 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9495d72-44b3-4811-8b94-0c3a497044bc None admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:56.087945 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9495d72-44b3-4811-8b94-0c3a497044bc None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:56.088247 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9495d72-44b3-4811-8b94-0c3a497044bc None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:56.089742 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9495d72-44b3-4811-8b94-0c3a497044bc None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:56.102350 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 323/648] 10.222.0.90 () {56 vars in 1148 bytes} [Mon Aug 29 21:20:55 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:56.103170 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9495d72-44b3-4811-8b94-0c3a497044bc None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:56.113770 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6addf09d-4929-4863-bd5e-a0a6cb3904b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:56.114808 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6addf09d-4929-4863-bd5e-a0a6cb3904b6 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:56.115270 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6addf09d-4929-4863-bd5e-a0a6cb3904b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:56.115740 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6addf09d-4929-4863-bd5e-a0a6cb3904b6 None admin] PATH_INFO: `/v3/users/glance` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:56.123782 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6addf09d-4929-4863-bd5e-a0a6cb3904b6 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:56.125591 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6addf09d-4929-4863-bd5e-a0a6cb3904b6 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:56.138425 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-6addf09d-4929-4863-bd5e-a0a6cb3904b6 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Aug 29 21:20:56.138425 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:56.138425 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:56.138425 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:56.138425 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:56.138425 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:56.138425 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:56.138425 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:56.138425 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:56.138425 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:56.138425 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:56.138425 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:56.138425 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 29 21:20:56.138425 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 29 21:20:56.138425 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 29 21:20:56.138425 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 29 21:20:56.139404 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:56.139404 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:56.139404 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 29 21:20:56.139404 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 21:20:56.139404 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 29 21:20:56.139404 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 21:20:56.139404 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:56.139404 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:56.139404 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:56.139404 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:56.139404 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:56.139404 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:56.139404 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:56.139404 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:56.139404 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:56.139404 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:56.139404 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:56.139404 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:56.141038 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:56.141038 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:56.141038 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 29 21:20:56.141038 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 29 21:20:56.141038 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 29 21:20:56.141038 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 29 21:20:56.141038 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 29 21:20:56.141038 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 29 21:20:56.141038 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Aug 29 21:20:56.141038 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:20:56.147978 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 326/649] 10.222.0.90 () {56 vars in 1099 bytes} [Mon Aug 29 21:20:55 2022] GET /identity/v3/users/glance => generated 84 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:20:56.147978 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6addf09d-4929-4863-bd5e-a0a6cb3904b6 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:56.149203 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:56.212759 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9495d72-44b3-4811-8b94-0c3a497044bc None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:56.247450 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dbf4965-b7ef-424f-b4ab-d6a6461a9f68 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:56.248075 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dbf4965-b7ef-424f-b4ab-d6a6461a9f68 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:56.248354 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dbf4965-b7ef-424f-b4ab-d6a6461a9f68 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:56.248626 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dbf4965-b7ef-424f-b4ab-d6a6461a9f68 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:56.253305 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6addf09d-4929-4863-bd5e-a0a6cb3904b6 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:56.256205 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dbf4965-b7ef-424f-b4ab-d6a6461a9f68 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:56.257629 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dbf4965-b7ef-424f-b4ab-d6a6461a9f68 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:56.269425 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-6dbf4965-b7ef-424f-b4ab-d6a6461a9f68 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Aug 29 21:20:56.269425 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:56.269425 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:56.269425 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:56.269425 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:56.269425 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:56.269425 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:56.269425 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:56.269425 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:56.269425 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:56.269425 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:56.269425 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:56.269425 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 29 21:20:56.269425 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 29 21:20:56.269425 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 29 21:20:56.269425 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 29 21:20:56.270419 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:56.270419 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:56.270419 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 29 21:20:56.270419 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 21:20:56.270419 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 29 21:20:56.270419 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 21:20:56.270419 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:56.270419 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:56.270419 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:56.270419 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:56.270419 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:56.270419 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:56.270419 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:56.270419 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:56.270419 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:56.270419 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:56.270419 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:56.270419 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:56.271900 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:56.271900 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:56.271900 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 29 21:20:56.271900 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 29 21:20:56.271900 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 29 21:20:56.271900 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 29 21:20:56.271900 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 29 21:20:56.271900 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 29 21:20:56.271900 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Aug 29 21:20:56.271900 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:20:56.271900 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 324/650] 10.222.0.90 () {56 vars in 1099 bytes} [Mon Aug 29 21:20:56 2022] GET /identity/v3/users/cinder => generated 84 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:20:56.271900 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dbf4965-b7ef-424f-b4ab-d6a6461a9f68 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:56.292836 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67ac2931-c41a-4d8d-aed0-5d7fe9133b45 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:56.293447 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67ac2931-c41a-4d8d-aed0-5d7fe9133b45 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:56.293723 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67ac2931-c41a-4d8d-aed0-5d7fe9133b45 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:56.294029 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67ac2931-c41a-4d8d-aed0-5d7fe9133b45 None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:56.294683 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67ac2931-c41a-4d8d-aed0-5d7fe9133b45 None admin] RBAC: Adding query filter params (name=service) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:56.294984 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67ac2931-c41a-4d8d-aed0-5d7fe9133b45 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:56.296328 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67ac2931-c41a-4d8d-aed0-5d7fe9133b45 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:56.304336 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 327/651] 10.222.0.90 () {56 vars in 1103 bytes} [Mon Aug 29 21:20:56 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:56.305056 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67ac2931-c41a-4d8d-aed0-5d7fe9133b45 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:56.352494 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:56.379365 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dbf4965-b7ef-424f-b4ab-d6a6461a9f68 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:56.380924 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:56.414936 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebd4f7c8-a800-464b-8aa8-a263a0077fac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:56.415542 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebd4f7c8-a800-464b-8aa8-a263a0077fac None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:56.415825 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebd4f7c8-a800-464b-8aa8-a263a0077fac None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:56.416099 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebd4f7c8-a800-464b-8aa8-a263a0077fac None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:56.416226 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67ac2931-c41a-4d8d-aed0-5d7fe9133b45 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:56.416827 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebd4f7c8-a800-464b-8aa8-a263a0077fac None admin] RBAC: Adding query filter params (role.id=a94bdb4c5796428baf63644c062dfcb2, scope.project.id=ec03e112e3d547fbab6faf7b485c3fca, user.id=a35b5b7dea7b421da056f063d2436ffb) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:56.417132 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebd4f7c8-a800-464b-8aa8-a263a0077fac None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:56.418598 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebd4f7c8-a800-464b-8aa8-a263a0077fac None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:56.424214 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 325/652] 10.222.0.90 () {56 vars in 1374 bytes} [Mon Aug 29 21:20:56 2022] GET /identity/v3/role_assignments?user.id=a35b5b7dea7b421da056f063d2436ffb&scope.project.id=ec03e112e3d547fbab6faf7b485c3fca&role.id=a94bdb4c5796428baf63644c062dfcb2 => generated 608 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:56.425578 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-031f24e2-6a9f-4f03-8505-b5deec381629 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:56.425887 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-031f24e2-6a9f-4f03-8505-b5deec381629 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:56.426168 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-031f24e2-6a9f-4f03-8505-b5deec381629 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:56.427379 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 326/653] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:20:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:20:56.428079 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-031f24e2-6a9f-4f03-8505-b5deec381629 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:56.469697 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8310c9e3-7908-491f-8095-0f6ab91bf2bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:56.470745 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8310c9e3-7908-491f-8095-0f6ab91bf2bc None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:56.471480 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8310c9e3-7908-491f-8095-0f6ab91bf2bc None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:56.472502 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8310c9e3-7908-491f-8095-0f6ab91bf2bc None admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:56.473284 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8310c9e3-7908-491f-8095-0f6ab91bf2bc None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:56.473596 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8310c9e3-7908-491f-8095-0f6ab91bf2bc None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:56.475140 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8310c9e3-7908-491f-8095-0f6ab91bf2bc None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:56.495191 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 328/654] 10.222.0.90 () {56 vars in 1137 bytes} [Mon Aug 29 21:20:56 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:56.496062 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8310c9e3-7908-491f-8095-0f6ab91bf2bc None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:56.559884 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-031f24e2-6a9f-4f03-8505-b5deec381629 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:56.602708 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d5ffce3-e5e6-4e3d-b4b2-256aeb4b1928 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:56.603450 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d5ffce3-e5e6-4e3d-b4b2-256aeb4b1928 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:56.603729 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d5ffce3-e5e6-4e3d-b4b2-256aeb4b1928 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:56.604005 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d5ffce3-e5e6-4e3d-b4b2-256aeb4b1928 None admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:56.604849 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d5ffce3-e5e6-4e3d-b4b2-256aeb4b1928 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:56.605117 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d5ffce3-e5e6-4e3d-b4b2-256aeb4b1928 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:56.606589 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d5ffce3-e5e6-4e3d-b4b2-256aeb4b1928 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:56.619018 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:56.628088 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8310c9e3-7908-491f-8095-0f6ab91bf2bc None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:56.628668 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 327/655] 10.222.0.90 () {56 vars in 1137 bytes} [Mon Aug 29 21:20:56 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:56.629973 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39bf70cf-a067-4db0-b7e2-6a345a4a5c40 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:56.630269 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39bf70cf-a067-4db0-b7e2-6a345a4a5c40 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:56.630553 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39bf70cf-a067-4db0-b7e2-6a345a4a5c40 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:56.657752 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-39bf70cf-a067-4db0-b7e2-6a345a4a5c40 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:20:56.659619 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-39bf70cf-a067-4db0-b7e2-6a345a4a5c40 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:20:56.667040 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-948d21ff-477b-43dc-84a7-2aa3adebef13 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:56.667712 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-948d21ff-477b-43dc-84a7-2aa3adebef13 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:56.667961 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-948d21ff-477b-43dc-84a7-2aa3adebef13 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:56.668235 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-948d21ff-477b-43dc-84a7-2aa3adebef13 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:56.679517 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-948d21ff-477b-43dc-84a7-2aa3adebef13 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:56.679792 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:56.680956 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-948d21ff-477b-43dc-84a7-2aa3adebef13 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:56.697567 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-948d21ff-477b-43dc-84a7-2aa3adebef13 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:20:56.697567 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:56.697567 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:56.697567 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:56.697567 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:56.697567 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:56.697567 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:56.697567 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:56.697567 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:56.697567 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:56.697567 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:56.697567 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:56.697567 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 21:20:56.697567 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 21:20:56.697567 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 21:20:56.697567 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 21:20:56.698642 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:56.698642 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:56.698642 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:56.698642 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:56.698642 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:56.698642 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:56.698642 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:56.698642 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:56.698642 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:56.698642 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:56.698642 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:56.698642 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:56.698642 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:56.698642 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:56.698642 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:56.698642 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:56.698642 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 21:20:56.704603 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 21:20:56.704603 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:20:56.704603 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:20:56.704603 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 329/656] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:20:56 2022] GET /identity/v3/domains/Default => generated 87 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:20:56.704603 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-948d21ff-477b-43dc-84a7-2aa3adebef13 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:56.739216 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 328/657] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:20:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 21:20:56.749966 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39bf70cf-a067-4db0-b7e2-6a345a4a5c40 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:56.813535 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-948d21ff-477b-43dc-84a7-2aa3adebef13 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:56.865560 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6db32a81-d6ee-4737-a0f5-4f0e08f58d2c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:56.866434 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6db32a81-d6ee-4737-a0f5-4f0e08f58d2c None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:56.866898 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6db32a81-d6ee-4737-a0f5-4f0e08f58d2c None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:56.867403 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6db32a81-d6ee-4737-a0f5-4f0e08f58d2c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:56.872487 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39bf70cf-a067-4db0-b7e2-6a345a4a5c40 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:56.893229 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:56.911673 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70c5c586-93c9-4398-b6ef-b2e01d9bae37 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:56.912565 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70c5c586-93c9-4398-b6ef-b2e01d9bae37 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:56.913050 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70c5c586-93c9-4398-b6ef-b2e01d9bae37 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:56.913496 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70c5c586-93c9-4398-b6ef-b2e01d9bae37 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:56.958514 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6db32a81-d6ee-4737-a0f5-4f0e08f58d2c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:56.960318 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6db32a81-d6ee-4737-a0f5-4f0e08f58d2c None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:56.962568 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:56.994893 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70c5c586-93c9-4398-b6ef-b2e01d9bae37 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:56.996255 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70c5c586-93c9-4398-b6ef-b2e01d9bae37 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:57.083088 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 330/658] 10.222.0.90 () {58 vars in 1282 bytes} [Mon Aug 29 21:20:56 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 378 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 29 21:20:57.083088 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6db32a81-d6ee-4737-a0f5-4f0e08f58d2c None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:57.108906 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 329/659] 10.222.0.90 () {58 vars in 1282 bytes} [Mon Aug 29 21:20:56 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 362 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 29 21:20:57.108906 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70c5c586-93c9-4398-b6ef-b2e01d9bae37 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:57.146518 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:57.162251 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:57.204421 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6db32a81-d6ee-4737-a0f5-4f0e08f58d2c None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:57.206276 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70c5c586-93c9-4398-b6ef-b2e01d9bae37 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:57.240857 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16af6fd5-0e15-4af0-a556-24d3dcc55aa9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:57.241295 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-816c5922-84ab-48d0-8bd7-857b535b3f2c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:57.241509 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16af6fd5-0e15-4af0-a556-24d3dcc55aa9 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:57.241774 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16af6fd5-0e15-4af0-a556-24d3dcc55aa9 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:57.241978 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-816c5922-84ab-48d0-8bd7-857b535b3f2c None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:57.242074 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16af6fd5-0e15-4af0-a556-24d3dcc55aa9 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:57.242235 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-816c5922-84ab-48d0-8bd7-857b535b3f2c None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:57.242496 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-816c5922-84ab-48d0-8bd7-857b535b3f2c None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:57.243332 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-816c5922-84ab-48d0-8bd7-857b535b3f2c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:57.243639 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-816c5922-84ab-48d0-8bd7-857b535b3f2c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:57.245243 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-816c5922-84ab-48d0-8bd7-857b535b3f2c None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:57.252178 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16af6fd5-0e15-4af0-a556-24d3dcc55aa9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 21:20:57.258614 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-16af6fd5-0e15-4af0-a556-24d3dcc55aa9 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 29 21:20:57.258614 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:57.258614 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:57.258614 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:57.258614 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:57.258614 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:57.258614 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:57.258614 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:57.258614 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:57.258614 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:57.258614 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:57.258614 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:57.258614 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 21:20:57.258614 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 21:20:57.258614 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 21:20:57.258614 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 21:20:57.259880 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 21:20:57.259880 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 21:20:57.259880 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 21:20:57.259880 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 21:20:57.259880 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:57.259880 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:57.259880 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:57.259880 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:57.259880 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:57.259880 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:57.259880 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:57.259880 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:57.259880 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:57.259880 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:57.259880 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:57.259880 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:57.260745 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:57.260745 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:57.260745 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:57.260745 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:57.260745 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 21:20:57.260745 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 21:20:57.260745 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 21:20:57.260745 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 21:20:57.260745 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 21:20:57.260745 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 21:20:57.260745 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 29 21:20:57.260745 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:20:57.260745 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 330/660] 10.222.0.90 () {56 vars in 1102 bytes} [Mon Aug 29 21:20:57 2022] GET /identity/v3/roles/service => generated 85 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:20:57.260745 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16af6fd5-0e15-4af0-a556-24d3dcc55aa9 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:57.267562 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 331/661] 10.222.0.90 () {56 vars in 1109 bytes} [Mon Aug 29 21:20:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:57.267562 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-816c5922-84ab-48d0-8bd7-857b535b3f2c None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:57.373265 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-816c5922-84ab-48d0-8bd7-857b535b3f2c None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:57.373949 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16af6fd5-0e15-4af0-a556-24d3dcc55aa9 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:57.382360 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:57.392686 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:57.409598 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65304f45-10e7-4573-be3b-7203c05f3f85 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:57.410259 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65304f45-10e7-4573-be3b-7203c05f3f85 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:57.410539 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65304f45-10e7-4573-be3b-7203c05f3f85 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:57.410813 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65304f45-10e7-4573-be3b-7203c05f3f85 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:57.418200 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d00ae1c5-df28-4732-a7f8-8fa3fdf042b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:57.418832 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d00ae1c5-df28-4732-a7f8-8fa3fdf042b7 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:57.419380 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d00ae1c5-df28-4732-a7f8-8fa3fdf042b7 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:57.419380 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d00ae1c5-df28-4732-a7f8-8fa3fdf042b7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:57.422985 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65304f45-10e7-4573-be3b-7203c05f3f85 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:57.424554 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65304f45-10e7-4573-be3b-7203c05f3f85 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:57.430650 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d00ae1c5-df28-4732-a7f8-8fa3fdf042b7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:57.432395 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d00ae1c5-df28-4732-a7f8-8fa3fdf042b7 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:57.442947 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-65304f45-10e7-4573-be3b-7203c05f3f85 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:20:57.442947 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:57.442947 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:57.442947 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:57.442947 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:57.442947 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:57.442947 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:57.442947 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:57.442947 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:57.442947 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:57.442947 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:57.442947 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:57.442947 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 21:20:57.442947 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 21:20:57.442947 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 21:20:57.442947 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 21:20:57.444270 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:57.444270 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:57.444270 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:57.444270 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:57.444270 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:57.444270 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:57.444270 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:57.444270 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:57.444270 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:57.444270 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:57.444270 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:57.444270 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:57.444270 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:57.444270 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:57.444270 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:57.444270 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:57.444270 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 21:20:57.445329 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 21:20:57.445329 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:20:57.445329 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:20:57.445329 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 332/662] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:20:57 2022] GET /identity/v3/domains/Default => generated 87 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:20:57.445329 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65304f45-10e7-4573-be3b-7203c05f3f85 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:57.452133 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-d00ae1c5-df28-4732-a7f8-8fa3fdf042b7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:20:57.452133 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:57.452133 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:57.452133 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:57.452133 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:57.452133 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:57.452133 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:57.452133 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:57.452133 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:57.452133 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:57.452133 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:57.452133 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:57.452133 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 21:20:57.452133 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 21:20:57.452133 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 21:20:57.452133 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 21:20:57.453195 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:57.453195 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:57.453195 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:57.453195 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:57.453195 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:57.453195 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:57.453195 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:57.453195 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:57.453195 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:57.453195 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:57.453195 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:57.453195 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:57.453195 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:57.453195 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:57.453195 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:57.453195 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:57.453195 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 21:20:57.454833 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 21:20:57.454833 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:20:57.454833 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:20:57.454833 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 331/663] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:20:57 2022] GET /identity/v3/domains/Default => generated 87 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:20:57.454833 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d00ae1c5-df28-4732-a7f8-8fa3fdf042b7 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:57.564410 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65304f45-10e7-4573-be3b-7203c05f3f85 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:57.565675 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d00ae1c5-df28-4732-a7f8-8fa3fdf042b7 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:57.603174 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59bb257f-1de5-48ac-a7c3-729eb5e8ac24 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:57.603787 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59bb257f-1de5-48ac-a7c3-729eb5e8ac24 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:57.604076 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59bb257f-1de5-48ac-a7c3-729eb5e8ac24 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:57.604340 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59bb257f-1de5-48ac-a7c3-729eb5e8ac24 None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:57.604939 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-914e560c-a4de-4ca4-887f-853ae7b6ee5b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:57.605060 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59bb257f-1de5-48ac-a7c3-729eb5e8ac24 None admin] RBAC: Adding query filter params (name=service) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:57.605290 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59bb257f-1de5-48ac-a7c3-729eb5e8ac24 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:57.605596 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-914e560c-a4de-4ca4-887f-853ae7b6ee5b None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:57.605882 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-914e560c-a4de-4ca4-887f-853ae7b6ee5b None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:57.606157 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-914e560c-a4de-4ca4-887f-853ae7b6ee5b None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:57.606709 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59bb257f-1de5-48ac-a7c3-729eb5e8ac24 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:57.614760 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-914e560c-a4de-4ca4-887f-853ae7b6ee5b None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:57.621089 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 332/664] 10.222.0.90 () {56 vars in 1103 bytes} [Mon Aug 29 21:20:57 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:57.621243 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-914e560c-a4de-4ca4-887f-853ae7b6ee5b None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:57.621344 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59bb257f-1de5-48ac-a7c3-729eb5e8ac24 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:57.632272 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-914e560c-a4de-4ca4-887f-853ae7b6ee5b None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Aug 29 21:20:57.632272 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:57.632272 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:57.632272 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:57.632272 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:57.632272 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:57.632272 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:57.632272 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:57.632272 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:57.632272 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:57.632272 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:57.632272 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:57.632272 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 29 21:20:57.632272 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 29 21:20:57.632272 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 29 21:20:57.632272 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 29 21:20:57.633263 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:57.633263 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:57.633263 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 29 21:20:57.633263 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 21:20:57.633263 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 29 21:20:57.633263 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 21:20:57.633263 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:57.633263 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:57.633263 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:57.633263 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:57.633263 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:57.633263 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:57.633263 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:57.633263 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:57.633263 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:57.633263 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:57.633263 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:57.633263 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:57.634786 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:57.634786 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:57.634786 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 29 21:20:57.634786 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 29 21:20:57.634786 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 29 21:20:57.634786 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 29 21:20:57.634786 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 29 21:20:57.634786 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 29 21:20:57.634786 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Aug 29 21:20:57.634786 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:20:57.634786 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 333/665] 10.222.0.90 () {56 vars in 1102 bytes} [Mon Aug 29 21:20:57 2022] GET /identity/v3/users/neutron => generated 85 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:20:57.634786 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-914e560c-a4de-4ca4-887f-853ae7b6ee5b None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:57.656367 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:57.659978 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:57.733540 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59bb257f-1de5-48ac-a7c3-729eb5e8ac24 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:57.746734 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-914e560c-a4de-4ca4-887f-853ae7b6ee5b None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:57.769363 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-030ba964-df51-4356-9758-a82d1f4b5926 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:57.770244 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-030ba964-df51-4356-9758-a82d1f4b5926 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:57.770648 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-030ba964-df51-4356-9758-a82d1f4b5926 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:57.771036 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-030ba964-df51-4356-9758-a82d1f4b5926 None admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:57.771797 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-030ba964-df51-4356-9758-a82d1f4b5926 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:57.772194 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-030ba964-df51-4356-9758-a82d1f4b5926 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:57.773560 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-030ba964-df51-4356-9758-a82d1f4b5926 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:57.782736 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d7a8f16-0c6d-489b-a386-13a5316c2fa8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:57.783532 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d7a8f16-0c6d-489b-a386-13a5316c2fa8 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:57.784660 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d7a8f16-0c6d-489b-a386-13a5316c2fa8 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:57.785135 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d7a8f16-0c6d-489b-a386-13a5316c2fa8 None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:57.785892 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 333/666] 10.222.0.90 () {56 vars in 1109 bytes} [Mon Aug 29 21:20:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:57.786102 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d7a8f16-0c6d-489b-a386-13a5316c2fa8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:57.786555 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d7a8f16-0c6d-489b-a386-13a5316c2fa8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:57.786715 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-030ba964-df51-4356-9758-a82d1f4b5926 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:57.788236 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d7a8f16-0c6d-489b-a386-13a5316c2fa8 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:57.802229 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 334/667] 10.222.0.90 () {56 vars in 1109 bytes} [Mon Aug 29 21:20:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:57.803125 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d7a8f16-0c6d-489b-a386-13a5316c2fa8 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:57.898900 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:57.899556 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:57.917581 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-030ba964-df51-4356-9758-a82d1f4b5926 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:57.921900 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d7a8f16-0c6d-489b-a386-13a5316c2fa8 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:57.953958 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af29e35f-73fd-4a7b-8815-d9c33c581252 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:57.954556 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af29e35f-73fd-4a7b-8815-d9c33c581252 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:57.954855 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af29e35f-73fd-4a7b-8815-d9c33c581252 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:57.955098 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af29e35f-73fd-4a7b-8815-d9c33c581252 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:57.957770 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58de2f7c-abb6-44e8-bb57-3673c466d078 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:57.958394 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58de2f7c-abb6-44e8-bb57-3673c466d078 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:57.958679 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58de2f7c-abb6-44e8-bb57-3673c466d078 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:57.959242 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58de2f7c-abb6-44e8-bb57-3673c466d078 None admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:57.959653 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58de2f7c-abb6-44e8-bb57-3673c466d078 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:57.959956 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58de2f7c-abb6-44e8-bb57-3673c466d078 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:57.961372 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58de2f7c-abb6-44e8-bb57-3673c466d078 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:57.966970 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af29e35f-73fd-4a7b-8815-d9c33c581252 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:57.968416 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af29e35f-73fd-4a7b-8815-d9c33c581252 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:57.980653 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 335/668] 10.222.0.90 () {56 vars in 1139 bytes} [Mon Aug 29 21:20:57 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:57.981746 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58de2f7c-abb6-44e8-bb57-3673c466d078 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:57.984923 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-af29e35f-73fd-4a7b-8815-d9c33c581252 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:20:57.984923 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:57.984923 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:57.984923 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:57.984923 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:57.984923 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:57.984923 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:57.984923 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:57.984923 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:57.984923 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:57.984923 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:57.984923 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:57.984923 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 21:20:57.984923 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 21:20:57.984923 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 21:20:57.984923 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 21:20:57.985987 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:57.985987 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:57.985987 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:57.985987 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:57.985987 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:57.985987 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:57.985987 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:57.985987 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:57.985987 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:57.985987 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:57.985987 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:57.985987 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:57.985987 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:57.985987 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:57.985987 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:57.985987 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:57.985987 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 21:20:57.987283 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 21:20:57.987283 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:20:57.987283 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:20:57.990909 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 334/669] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:20:57 2022] GET /identity/v3/domains/Default => generated 87 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:20:57.992031 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af29e35f-73fd-4a7b-8815-d9c33c581252 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:58.095780 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58de2f7c-abb6-44e8-bb57-3673c466d078 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:58.099430 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af29e35f-73fd-4a7b-8815-d9c33c581252 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:58.135516 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0837a69-fdff-4273-aa0e-32b9010daa31 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:58.136173 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0837a69-fdff-4273-aa0e-32b9010daa31 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:58.136451 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0837a69-fdff-4273-aa0e-32b9010daa31 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:58.136722 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0837a69-fdff-4273-aa0e-32b9010daa31 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:58.137367 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9716a572-f58d-416d-98cd-e7f78b063696 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:58.138103 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9716a572-f58d-416d-98cd-e7f78b063696 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:58.138382 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9716a572-f58d-416d-98cd-e7f78b063696 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:58.138813 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9716a572-f58d-416d-98cd-e7f78b063696 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:58.144311 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0837a69-fdff-4273-aa0e-32b9010daa31 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:58.146024 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0837a69-fdff-4273-aa0e-32b9010daa31 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:58.147030 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9716a572-f58d-416d-98cd-e7f78b063696 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:58.147706 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:58.148868 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9716a572-f58d-416d-98cd-e7f78b063696 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:58.151953 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:58.159465 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-f0837a69-fdff-4273-aa0e-32b9010daa31 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 21:20:58.159465 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:58.159465 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:58.159465 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:58.159465 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:58.159465 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:58.159465 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:58.159465 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:58.159465 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:58.159465 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:58.159465 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:58.159465 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:58.159465 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 29 21:20:58.159465 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 29 21:20:58.159465 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 29 21:20:58.160405 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 29 21:20:58.160405 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:58.160405 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:58.160405 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:58.160405 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:58.160405 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:58.160405 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:58.160405 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:58.160405 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:58.160405 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:58.160405 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:58.160405 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:58.160405 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:58.160405 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:58.160405 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:58.160405 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:58.160405 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:58.166636 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 29 21:20:58.166636 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 29 21:20:58.166636 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 29 21:20:58.166636 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 29 21:20:58.166636 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 29 21:20:58.166636 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 29 21:20:58.166636 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 21:20:58.166636 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:20:58.166636 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 336/670] 10.222.0.90 () {56 vars in 1111 bytes} [Mon Aug 29 21:20:57 2022] GET /identity/v3/projects/service => generated 88 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:20:58.167176 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-9716a572-f58d-416d-98cd-e7f78b063696 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 21:20:58.167176 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:58.167176 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:58.167176 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:58.167176 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:58.167176 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:58.167176 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:58.167176 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:58.167176 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:58.167176 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:58.167176 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:58.167176 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:58.167176 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 29 21:20:58.167176 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 29 21:20:58.167176 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 29 21:20:58.168101 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 29 21:20:58.168101 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:58.168101 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:58.168101 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:58.168101 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:58.168101 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:58.168101 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:58.168101 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:58.168101 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:58.168101 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:58.168101 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:58.168101 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:58.168101 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:58.168101 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:58.168101 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:58.168101 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:58.168101 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:58.169013 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 29 21:20:58.169013 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 29 21:20:58.169013 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 29 21:20:58.169013 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 29 21:20:58.169013 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 29 21:20:58.169013 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 29 21:20:58.169013 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 21:20:58.169013 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:20:58.169451 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0837a69-fdff-4273-aa0e-32b9010daa31 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:58.169525 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 335/671] 10.222.0.90 () {56 vars in 1111 bytes} [Mon Aug 29 21:20:57 2022] GET /identity/v3/projects/service => generated 88 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:20:58.169525 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9716a572-f58d-416d-98cd-e7f78b063696 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:58.275092 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0837a69-fdff-4273-aa0e-32b9010daa31 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:58.276090 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9716a572-f58d-416d-98cd-e7f78b063696 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:58.312091 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae556320-d212-4845-a984-4d6d1b436ef0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:58.312705 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae556320-d212-4845-a984-4d6d1b436ef0 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:58.312988 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae556320-d212-4845-a984-4d6d1b436ef0 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:58.313264 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae556320-d212-4845-a984-4d6d1b436ef0 None admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:58.313946 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae556320-d212-4845-a984-4d6d1b436ef0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:58.314255 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae556320-d212-4845-a984-4d6d1b436ef0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:58.314630 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1829d7f4-0116-4321-8ee8-7aabf1a363d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:58.315842 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1829d7f4-0116-4321-8ee8-7aabf1a363d5 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:58.316116 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1829d7f4-0116-4321-8ee8-7aabf1a363d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:58.316213 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae556320-d212-4845-a984-4d6d1b436ef0 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:58.316379 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1829d7f4-0116-4321-8ee8-7aabf1a363d5 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:58.333189 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 336/672] 10.222.0.90 () {56 vars in 1109 bytes} [Mon Aug 29 21:20:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:58.333189 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae556320-d212-4845-a984-4d6d1b436ef0 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:58.392929 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:58.403362 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:58.407336 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1829d7f4-0116-4321-8ee8-7aabf1a363d5 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:58.408569 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1829d7f4-0116-4321-8ee8-7aabf1a363d5 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:58.441901 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae556320-d212-4845-a984-4d6d1b436ef0 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:58.476510 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-528a736f-6d9a-48bd-a74b-7c7ddee11cc1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:58.477119 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-528a736f-6d9a-48bd-a74b-7c7ddee11cc1 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:58.477398 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-528a736f-6d9a-48bd-a74b-7c7ddee11cc1 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:58.477669 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-528a736f-6d9a-48bd-a74b-7c7ddee11cc1 None admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:58.478391 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-528a736f-6d9a-48bd-a74b-7c7ddee11cc1 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:58.478696 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-528a736f-6d9a-48bd-a74b-7c7ddee11cc1 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:58.480131 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-528a736f-6d9a-48bd-a74b-7c7ddee11cc1 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:58.492289 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 337/673] 10.222.0.90 () {56 vars in 1148 bytes} [Mon Aug 29 21:20:58 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:58.493089 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-528a736f-6d9a-48bd-a74b-7c7ddee11cc1 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:58.513512 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 337/674] 10.222.0.90 () {58 vars in 1282 bytes} [Mon Aug 29 21:20:58 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 352 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 29 21:20:58.514389 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1829d7f4-0116-4321-8ee8-7aabf1a363d5 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:58.594424 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-528a736f-6d9a-48bd-a74b-7c7ddee11cc1 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:58.607282 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:58.611265 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1829d7f4-0116-4321-8ee8-7aabf1a363d5 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:58.615099 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:58.630621 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2836e7ad-2e06-40ad-ae65-7e1cac097a47 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:58.631236 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2836e7ad-2e06-40ad-ae65-7e1cac097a47 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:58.631520 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2836e7ad-2e06-40ad-ae65-7e1cac097a47 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:58.631796 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2836e7ad-2e06-40ad-ae65-7e1cac097a47 None admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:58.632492 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2836e7ad-2e06-40ad-ae65-7e1cac097a47 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:58.632800 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2836e7ad-2e06-40ad-ae65-7e1cac097a47 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:58.634271 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2836e7ad-2e06-40ad-ae65-7e1cac097a47 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:58.646380 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 338/675] 10.222.0.90 () {56 vars in 1148 bytes} [Mon Aug 29 21:20:58 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:58.647556 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4787709f-afb5-4348-a3f6-294725ebb297 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:58.647843 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4787709f-afb5-4348-a3f6-294725ebb297 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:58.648119 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4787709f-afb5-4348-a3f6-294725ebb297 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:58.649313 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 339/676] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:20:58 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:20:58.650035 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4787709f-afb5-4348-a3f6-294725ebb297 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:58.651176 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c8ada66-08f6-40ae-923e-55ee7e96cac6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:58.651818 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c8ada66-08f6-40ae-923e-55ee7e96cac6 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:58.652096 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c8ada66-08f6-40ae-923e-55ee7e96cac6 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:58.652369 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c8ada66-08f6-40ae-923e-55ee7e96cac6 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:58.660826 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c8ada66-08f6-40ae-923e-55ee7e96cac6 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:58.662236 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c8ada66-08f6-40ae-923e-55ee7e96cac6 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:58.674085 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-3c8ada66-08f6-40ae-923e-55ee7e96cac6 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Aug 29 21:20:58.674085 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:58.674085 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:58.674085 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:58.674085 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:58.674085 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:58.674085 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:58.674085 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:58.674085 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:58.674085 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:58.674085 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:58.674085 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:58.674085 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 29 21:20:58.674085 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 29 21:20:58.674085 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 29 21:20:58.674085 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 29 21:20:58.675618 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:58.675618 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:58.675618 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 29 21:20:58.675618 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 21:20:58.675618 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 29 21:20:58.675618 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 21:20:58.675618 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:58.675618 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:58.675618 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:58.675618 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:58.675618 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:58.675618 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:58.675618 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:58.675618 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:58.675618 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:58.675618 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:58.675618 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:58.675618 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:58.677062 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:58.677062 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:58.677062 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 29 21:20:58.677062 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 29 21:20:58.677062 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 29 21:20:58.677062 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 29 21:20:58.677062 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 29 21:20:58.677062 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 29 21:20:58.677062 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Aug 29 21:20:58.677062 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:20:58.677062 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 338/677] 10.222.0.90 () {56 vars in 1096 bytes} [Mon Aug 29 21:20:58 2022] GET /identity/v3/users/swift => generated 83 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:20:58.677062 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c8ada66-08f6-40ae-923e-55ee7e96cac6 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:58.761577 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4787709f-afb5-4348-a3f6-294725ebb297 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:58.777874 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c8ada66-08f6-40ae-923e-55ee7e96cac6 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:58.798459 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1b262a3-e143-4eec-b1f7-b1398eed6f16 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:58.799076 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b262a3-e143-4eec-b1f7-b1398eed6f16 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:58.799429 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b262a3-e143-4eec-b1f7-b1398eed6f16 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:58.799937 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b262a3-e143-4eec-b1f7-b1398eed6f16 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:58.800797 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1b262a3-e143-4eec-b1f7-b1398eed6f16 None admin] RBAC: Adding query filter params (role.id=a94bdb4c5796428baf63644c062dfcb2, scope.project.id=ec03e112e3d547fbab6faf7b485c3fca, user.id=0dc226f7ddcb49ecacb51ce57fdfc11b) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:58.801105 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1b262a3-e143-4eec-b1f7-b1398eed6f16 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:58.802569 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1b262a3-e143-4eec-b1f7-b1398eed6f16 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:58.808372 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 340/678] 10.222.0.90 () {56 vars in 1374 bytes} [Mon Aug 29 21:20:58 2022] GET /identity/v3/role_assignments?user.id=0dc226f7ddcb49ecacb51ce57fdfc11b&scope.project.id=ec03e112e3d547fbab6faf7b485c3fca&role.id=a94bdb4c5796428baf63644c062dfcb2 => generated 608 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:58.809158 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1b262a3-e143-4eec-b1f7-b1398eed6f16 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:58.823490 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-977ff889-8298-40f7-b99c-9bf48b08d43f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:58.824163 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-977ff889-8298-40f7-b99c-9bf48b08d43f None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:58.824447 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-977ff889-8298-40f7-b99c-9bf48b08d43f None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:58.824724 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-977ff889-8298-40f7-b99c-9bf48b08d43f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:58.840024 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-977ff889-8298-40f7-b99c-9bf48b08d43f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:58.844432 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-977ff889-8298-40f7-b99c-9bf48b08d43f None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:58.862004 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-977ff889-8298-40f7-b99c-9bf48b08d43f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:20:58.862004 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:58.862004 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:58.862004 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:58.862004 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:58.862004 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:58.862004 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:58.862004 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:58.862004 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:58.862004 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:58.862004 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:58.862004 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:58.862004 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 21:20:58.862004 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 21:20:58.862004 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 21:20:58.862004 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 21:20:58.863177 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:58.863177 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:58.863177 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:58.863177 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:58.863177 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:58.863177 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:58.863177 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:58.863177 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:58.863177 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:58.863177 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:58.863177 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:58.863177 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:58.863177 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:58.863177 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:58.863177 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:58.863177 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:58.863177 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 21:20:58.864091 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 21:20:58.864091 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:20:58.864091 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:20:58.864091 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 339/679] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:20:58 2022] GET /identity/v3/domains/Default => generated 87 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:20:58.865419 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-434f9293-d2f8-4846-adfb-0d2bd1e1a5af None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:58.865419 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-434f9293-d2f8-4846-adfb-0d2bd1e1a5af None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:58.865419 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-434f9293-d2f8-4846-adfb-0d2bd1e1a5af None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:58.876400 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:58.885932 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-434f9293-d2f8-4846-adfb-0d2bd1e1a5af None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:20:58.887164 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-434f9293-d2f8-4846-adfb-0d2bd1e1a5af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:20:58.891697 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:58.944005 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1b262a3-e143-4eec-b1f7-b1398eed6f16 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:58.969840 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 340/680] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:20:58 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 21:20:58.970354 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-434f9293-d2f8-4846-adfb-0d2bd1e1a5af None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:58.989273 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-455768cf-2bd3-4d20-9aac-4775f17d0811 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:58.989877 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-455768cf-2bd3-4d20-9aac-4775f17d0811 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:58.990155 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-455768cf-2bd3-4d20-9aac-4775f17d0811 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:58.990425 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-455768cf-2bd3-4d20-9aac-4775f17d0811 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:58.991196 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-455768cf-2bd3-4d20-9aac-4775f17d0811 None admin] RBAC: Adding query filter params (role.id=a94bdb4c5796428baf63644c062dfcb2, scope.project.id=ec03e112e3d547fbab6faf7b485c3fca, user.id=c63a4eaf48ad43529f0aa328f2ebfd29) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:58.991503 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-455768cf-2bd3-4d20-9aac-4775f17d0811 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:58.993066 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-455768cf-2bd3-4d20-9aac-4775f17d0811 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:59.007588 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 341/681] 10.222.0.90 () {56 vars in 1374 bytes} [Mon Aug 29 21:20:58 2022] GET /identity/v3/role_assignments?user.id=c63a4eaf48ad43529f0aa328f2ebfd29&scope.project.id=ec03e112e3d547fbab6faf7b485c3fca&role.id=a94bdb4c5796428baf63644c062dfcb2 => generated 608 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:59.007588 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-455768cf-2bd3-4d20-9aac-4775f17d0811 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:59.123924 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-434f9293-d2f8-4846-adfb-0d2bd1e1a5af None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:59.144053 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-455768cf-2bd3-4d20-9aac-4775f17d0811 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:59.156256 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:59.184766 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09776e7c-7f16-4bd9-9e05-7e64c67c0008 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:59.185407 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09776e7c-7f16-4bd9-9e05-7e64c67c0008 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:59.185690 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09776e7c-7f16-4bd9-9e05-7e64c67c0008 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:59.186017 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09776e7c-7f16-4bd9-9e05-7e64c67c0008 None admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:59.186764 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09776e7c-7f16-4bd9-9e05-7e64c67c0008 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:59.187138 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09776e7c-7f16-4bd9-9e05-7e64c67c0008 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:59.189136 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09776e7c-7f16-4bd9-9e05-7e64c67c0008 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:59.194425 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34506648-bbde-4665-a5a0-c9b184bae0c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:59.194983 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34506648-bbde-4665-a5a0-c9b184bae0c0 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:59.195268 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34506648-bbde-4665-a5a0-c9b184bae0c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:59.195544 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34506648-bbde-4665-a5a0-c9b184bae0c0 None admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:59.196205 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34506648-bbde-4665-a5a0-c9b184bae0c0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:59.196512 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34506648-bbde-4665-a5a0-c9b184bae0c0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:59.197997 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34506648-bbde-4665-a5a0-c9b184bae0c0 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:59.199881 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:59.210863 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 341/682] 10.222.0.90 () {56 vars in 1135 bytes} [Mon Aug 29 21:20:58 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 241 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:59.213111 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 342/683] 10.222.0.90 () {56 vars in 1109 bytes} [Mon Aug 29 21:20:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:59.219483 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09776e7c-7f16-4bd9-9e05-7e64c67c0008 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:59.222622 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34506648-bbde-4665-a5a0-c9b184bae0c0 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:59.384773 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09776e7c-7f16-4bd9-9e05-7e64c67c0008 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:59.392175 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34506648-bbde-4665-a5a0-c9b184bae0c0 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:59.441204 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fee2bd7c-35ec-4733-84c9-773a09033ed0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:59.442312 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fee2bd7c-35ec-4733-84c9-773a09033ed0 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:59.442813 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fee2bd7c-35ec-4733-84c9-773a09033ed0 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:59.443278 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fee2bd7c-35ec-4733-84c9-773a09033ed0 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:59.449672 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f4cac95-32a0-4503-a645-20c941a28a33 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:59.450363 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f4cac95-32a0-4503-a645-20c941a28a33 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:59.450610 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f4cac95-32a0-4503-a645-20c941a28a33 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:59.450887 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f4cac95-32a0-4503-a645-20c941a28a33 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:59.460746 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f4cac95-32a0-4503-a645-20c941a28a33 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:59.462516 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f4cac95-32a0-4503-a645-20c941a28a33 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:59.479910 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-4f4cac95-32a0-4503-a645-20c941a28a33 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 21:20:59.479910 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:59.479910 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:59.479910 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:59.479910 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:59.479910 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:59.479910 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:59.479910 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:59.479910 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:59.479910 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:59.479910 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:59.479910 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:59.479910 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 29 21:20:59.479910 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 29 21:20:59.479910 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 29 21:20:59.481278 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 29 21:20:59.481278 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:59.481278 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:59.481278 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:59.481278 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:59.481278 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:59.481278 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:59.481278 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:59.481278 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:59.481278 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:59.481278 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:59.481278 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:59.481278 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:59.481278 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:59.481278 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:59.481278 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:59.481278 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:59.482825 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 29 21:20:59.482825 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 29 21:20:59.482825 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 29 21:20:59.482825 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 29 21:20:59.482825 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 29 21:20:59.482825 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 29 21:20:59.482825 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 21:20:59.482825 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:20:59.482825 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 343/684] 10.222.0.90 () {56 vars in 1111 bytes} [Mon Aug 29 21:20:59 2022] GET /identity/v3/projects/service => generated 88 bytes in 259 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:20:59.482825 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f4cac95-32a0-4503-a645-20c941a28a33 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:59.525568 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:59.526615 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:59.531112 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fee2bd7c-35ec-4733-84c9-773a09033ed0 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:59.533028 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fee2bd7c-35ec-4733-84c9-773a09033ed0 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:59.644680 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f4cac95-32a0-4503-a645-20c941a28a33 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:59.664549 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 342/685] 10.222.0.90 () {58 vars in 1282 bytes} [Mon Aug 29 21:20:59 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 446 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 29 21:20:59.666094 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fee2bd7c-35ec-4733-84c9-773a09033ed0 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:59.699198 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bdb6017-138a-4b97-8525-2abc89b9146b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:59.700319 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bdb6017-138a-4b97-8525-2abc89b9146b None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:59.700798 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bdb6017-138a-4b97-8525-2abc89b9146b None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:59.701266 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bdb6017-138a-4b97-8525-2abc89b9146b None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:59.714241 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bdb6017-138a-4b97-8525-2abc89b9146b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 21:20:59.719686 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-2bdb6017-138a-4b97-8525-2abc89b9146b None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 21:20:59.719686 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:59.719686 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:59.719686 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:59.719686 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:59.719686 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:59.719686 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:59.719686 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:59.719686 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:59.719686 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:59.719686 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:59.719686 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:59.719686 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 21:20:59.719686 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 21:20:59.719686 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 21:20:59.719686 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 21:20:59.720662 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 21:20:59.720662 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 21:20:59.720662 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 21:20:59.720662 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 21:20:59.720662 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:59.720662 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:59.720662 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:59.720662 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:59.720662 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:59.720662 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:59.720662 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:59.720662 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:59.720662 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:59.720662 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:59.720662 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:59.720662 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:59.721888 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:59.721888 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:59.721888 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:59.721888 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:59.721888 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 21:20:59.721888 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 21:20:59.721888 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 21:20:59.721888 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 21:20:59.721888 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 21:20:59.721888 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 21:20:59.721888 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 21:20:59.721888 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:20:59.729766 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 344/686] 10.222.0.90 () {56 vars in 1096 bytes} [Mon Aug 29 21:20:59 2022] GET /identity/v3/roles/admin => generated 83 bytes in 243 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:20:59.730999 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bdb6017-138a-4b97-8525-2abc89b9146b None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:59.781523 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:59.787353 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fee2bd7c-35ec-4733-84c9-773a09033ed0 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:59.824828 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9d4eec3-300b-416f-806c-4bb70384fb56 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:59.826500 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d4eec3-300b-416f-806c-4bb70384fb56 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:59.827054 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d4eec3-300b-416f-806c-4bb70384fb56 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:59.827620 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d4eec3-300b-416f-806c-4bb70384fb56 None admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:59.829080 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9d4eec3-300b-416f-806c-4bb70384fb56 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:20:59.829987 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9d4eec3-300b-416f-806c-4bb70384fb56 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:59.833072 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9d4eec3-300b-416f-806c-4bb70384fb56 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:59.841924 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:20:59.849689 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 343/687] 10.222.0.90 () {56 vars in 1148 bytes} [Mon Aug 29 21:20:59 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:20:59.851191 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9d4eec3-300b-416f-806c-4bb70384fb56 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:20:59.859569 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bdb6017-138a-4b97-8525-2abc89b9146b None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:20:59.906919 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-154435f1-746c-4264-ada4-7a47af419460 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:20:59.907942 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-154435f1-746c-4264-ada4-7a47af419460 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:20:59.908419 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-154435f1-746c-4264-ada4-7a47af419460 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:20:59.909617 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-154435f1-746c-4264-ada4-7a47af419460 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:20:59.921262 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-154435f1-746c-4264-ada4-7a47af419460 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:20:59.923276 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-154435f1-746c-4264-ada4-7a47af419460 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:20:59.939677 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-154435f1-746c-4264-ada4-7a47af419460 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:20:59.939677 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:20:59.939677 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:20:59.939677 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:20:59.939677 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:20:59.939677 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:20:59.939677 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:20:59.939677 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:20:59.939677 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:20:59.939677 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:20:59.939677 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:20:59.939677 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:20:59.939677 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 21:20:59.939677 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 21:20:59.939677 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 21:20:59.939677 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 21:20:59.940833 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:20:59.940833 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:20:59.940833 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:20:59.940833 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:20:59.940833 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:20:59.940833 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:20:59.940833 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:20:59.940833 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:20:59.940833 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:20:59.940833 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:20:59.940833 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:20:59.940833 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:20:59.940833 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:20:59.940833 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:20:59.940833 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:20:59.940833 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:20:59.940833 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 21:20:59.941755 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 21:20:59.941755 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:20:59.941755 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:20:59.943922 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 345/688] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:20:59 2022] GET /identity/v3/domains/Default => generated 87 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:20:59.944999 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-154435f1-746c-4264-ada4-7a47af419460 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:00.007297 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9d4eec3-300b-416f-806c-4bb70384fb56 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:00.047769 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-649cdcb6-5f98-4df1-9351-c3af572d595e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:00.048391 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-649cdcb6-5f98-4df1-9351-c3af572d595e None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:00.048669 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-649cdcb6-5f98-4df1-9351-c3af572d595e None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:00.048939 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-649cdcb6-5f98-4df1-9351-c3af572d595e None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:00.049968 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-649cdcb6-5f98-4df1-9351-c3af572d595e None admin] RBAC: Adding query filter params (name=admin) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:00.050268 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-649cdcb6-5f98-4df1-9351-c3af572d595e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:00.055269 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-649cdcb6-5f98-4df1-9351-c3af572d595e None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:00.069625 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 344/689] 10.222.0.90 () {56 vars in 1099 bytes} [Mon Aug 29 21:20:59 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:00.072011 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-649cdcb6-5f98-4df1-9351-c3af572d595e None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:00.077454 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-154435f1-746c-4264-ada4-7a47af419460 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:00.099793 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:00.116410 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a03e55e2-5d58-4ee9-bf7c-84248d136da9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:00.117425 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a03e55e2-5d58-4ee9-bf7c-84248d136da9 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:00.117967 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a03e55e2-5d58-4ee9-bf7c-84248d136da9 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:00.118434 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a03e55e2-5d58-4ee9-bf7c-84248d136da9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:00.119357 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a03e55e2-5d58-4ee9-bf7c-84248d136da9 None admin] RBAC: Adding query filter params (role.id=a94bdb4c5796428baf63644c062dfcb2, scope.project.id=ec03e112e3d547fbab6faf7b485c3fca, user.id=0e69b7f0c3574abc9de5fb580d22152a) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:00.119861 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a03e55e2-5d58-4ee9-bf7c-84248d136da9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:00.121487 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a03e55e2-5d58-4ee9-bf7c-84248d136da9 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:00.129160 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 346/690] 10.222.0.90 () {56 vars in 1374 bytes} [Mon Aug 29 21:20:59 2022] GET /identity/v3/role_assignments?user.id=0e69b7f0c3574abc9de5fb580d22152a&scope.project.id=ec03e112e3d547fbab6faf7b485c3fca&role.id=a94bdb4c5796428baf63644c062dfcb2 => generated 608 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:00.130339 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a03e55e2-5d58-4ee9-bf7c-84248d136da9 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:00.134447 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:00.185376 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-649cdcb6-5f98-4df1-9351-c3af572d595e None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:00.229078 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-607ea974-f3ed-4861-9f49-f94e0ad2a8f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:00.229698 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-607ea974-f3ed-4861-9f49-f94e0ad2a8f9 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:00.230021 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-607ea974-f3ed-4861-9f49-f94e0ad2a8f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:00.230294 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-607ea974-f3ed-4861-9f49-f94e0ad2a8f9 None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:00.230947 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-607ea974-f3ed-4861-9f49-f94e0ad2a8f9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:00.231247 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-607ea974-f3ed-4861-9f49-f94e0ad2a8f9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:00.237866 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-607ea974-f3ed-4861-9f49-f94e0ad2a8f9 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:00.248598 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 345/691] 10.222.0.90 () {56 vars in 1109 bytes} [Mon Aug 29 21:21:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:00.255035 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-607ea974-f3ed-4861-9f49-f94e0ad2a8f9 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:00.284848 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a03e55e2-5d58-4ee9-bf7c-84248d136da9 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:00.357514 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:00.381971 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-607ea974-f3ed-4861-9f49-f94e0ad2a8f9 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:00.412088 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13a916c4-f40e-4cff-a5dc-3db1259ec3f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:00.413208 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13a916c4-f40e-4cff-a5dc-3db1259ec3f6 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:00.413477 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13a916c4-f40e-4cff-a5dc-3db1259ec3f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:00.414192 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13a916c4-f40e-4cff-a5dc-3db1259ec3f6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:00.425815 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01f2d674-2a48-4441-82c2-767b6b69fb71 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:00.426481 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13a916c4-f40e-4cff-a5dc-3db1259ec3f6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:00.426995 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f2d674-2a48-4441-82c2-767b6b69fb71 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:00.427471 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f2d674-2a48-4441-82c2-767b6b69fb71 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:00.429049 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13a916c4-f40e-4cff-a5dc-3db1259ec3f6 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:00.429177 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f2d674-2a48-4441-82c2-767b6b69fb71 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:00.437304 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01f2d674-2a48-4441-82c2-767b6b69fb71 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:00.439202 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01f2d674-2a48-4441-82c2-767b6b69fb71 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:00.444876 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-13a916c4-f40e-4cff-a5dc-3db1259ec3f6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:21:00.444876 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:00.444876 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:00.444876 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:00.444876 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:00.444876 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:00.444876 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:00.444876 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:00.444876 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:00.444876 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:00.444876 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:00.444876 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:00.444876 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 21:21:00.444876 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 21:21:00.444876 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 21:21:00.444876 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 21:21:00.445984 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:00.445984 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:00.445984 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:00.445984 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:00.445984 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:00.445984 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:00.445984 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:00.445984 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:00.445984 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:00.445984 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:00.445984 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:00.445984 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:00.445984 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:00.445984 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:00.445984 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:00.445984 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:00.445984 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 21:21:00.446943 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 21:21:00.446943 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:21:00.446943 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:21:00.451040 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 347/692] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:21:00 2022] GET /identity/v3/domains/Default => generated 87 bytes in 316 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:21:00.452492 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13a916c4-f40e-4cff-a5dc-3db1259ec3f6 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:00.457712 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-01f2d674-2a48-4441-82c2-767b6b69fb71 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 21:21:00.457712 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:00.457712 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:00.457712 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:00.457712 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:00.457712 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:00.457712 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:00.457712 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:00.457712 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:00.457712 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:00.457712 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:00.457712 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:00.457712 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 29 21:21:00.457712 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 29 21:21:00.457712 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 29 21:21:00.458720 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 29 21:21:00.458720 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:00.458720 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:00.458720 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:00.458720 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:00.458720 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:00.458720 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:00.458720 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:00.458720 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:00.458720 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:00.458720 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:00.458720 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:00.458720 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:00.458720 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:00.458720 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:00.458720 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:00.458720 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:00.459663 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 29 21:21:00.459663 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 29 21:21:00.459663 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 29 21:21:00.459663 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 29 21:21:00.459663 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 29 21:21:00.459663 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 29 21:21:00.459663 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 21:21:00.459663 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:21:00.462286 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 346/693] 10.222.0.90 () {56 vars in 1111 bytes} [Mon Aug 29 21:21:00 2022] GET /identity/v3/projects/service => generated 88 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 29 21:21:00.468506 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01f2d674-2a48-4441-82c2-767b6b69fb71 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:00.517041 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:00.568389 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13a916c4-f40e-4cff-a5dc-3db1259ec3f6 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:00.588053 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01f2d674-2a48-4441-82c2-767b6b69fb71 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:00.605872 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43402d44-c84e-4482-aea5-9807cd15e815 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:00.606855 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43402d44-c84e-4482-aea5-9807cd15e815 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:00.607136 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43402d44-c84e-4482-aea5-9807cd15e815 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:00.607448 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43402d44-c84e-4482-aea5-9807cd15e815 None admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:00.608144 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43402d44-c84e-4482-aea5-9807cd15e815 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:00.608446 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43402d44-c84e-4482-aea5-9807cd15e815 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:00.609827 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43402d44-c84e-4482-aea5-9807cd15e815 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:00.625890 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 348/694] 10.222.0.90 () {56 vars in 1109 bytes} [Mon Aug 29 21:21:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:00.630518 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43402d44-c84e-4482-aea5-9807cd15e815 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:00.635803 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a47a759-6bfd-4fec-ace0-608cff00e9f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:00.636888 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a47a759-6bfd-4fec-ace0-608cff00e9f5 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:00.638268 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a47a759-6bfd-4fec-ace0-608cff00e9f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:00.638780 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a47a759-6bfd-4fec-ace0-608cff00e9f5 None admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:00.640654 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a47a759-6bfd-4fec-ace0-608cff00e9f5 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:00.642385 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a47a759-6bfd-4fec-ace0-608cff00e9f5 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:00.645732 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a47a759-6bfd-4fec-ace0-608cff00e9f5 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:00.656327 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:00.666148 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 347/695] 10.222.0.90 () {56 vars in 1148 bytes} [Mon Aug 29 21:21:00 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:21:00.668238 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a47a759-6bfd-4fec-ace0-608cff00e9f5 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:00.757953 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43402d44-c84e-4482-aea5-9807cd15e815 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:00.769480 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:00.809221 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0b473aa-edd0-4017-83cc-b45a12b1a2e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:00.809901 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0b473aa-edd0-4017-83cc-b45a12b1a2e9 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:00.810232 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0b473aa-edd0-4017-83cc-b45a12b1a2e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:00.810440 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0b473aa-edd0-4017-83cc-b45a12b1a2e9 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:00.821019 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a47a759-6bfd-4fec-ace0-608cff00e9f5 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:00.827118 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0b473aa-edd0-4017-83cc-b45a12b1a2e9 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:00.828588 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0b473aa-edd0-4017-83cc-b45a12b1a2e9 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:00.848156 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-a0b473aa-edd0-4017-83cc-b45a12b1a2e9 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 29 21:21:00.848156 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:00.848156 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:00.848156 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:00.848156 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:00.848156 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:00.848156 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:00.848156 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:00.848156 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:00.848156 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:00.848156 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:00.848156 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:00.848156 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 29 21:21:00.848156 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 29 21:21:00.848156 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 29 21:21:00.848156 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 29 21:21:00.849188 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:00.849188 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:00.849188 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 29 21:21:00.849188 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 21:21:00.849188 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 29 21:21:00.849188 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 21:21:00.849188 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:00.849188 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:00.849188 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:00.849188 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:00.849188 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:00.849188 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:00.849188 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:00.849188 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:00.849188 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:00.849188 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:00.849188 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:00.849188 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:00.850200 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:00.850200 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:00.850200 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 29 21:21:00.850200 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 29 21:21:00.850200 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 29 21:21:00.850200 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 29 21:21:00.850200 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 29 21:21:00.850200 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 29 21:21:00.850200 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 29 21:21:00.850200 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:21:00.858230 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 349/696] 10.222.0.90 () {56 vars in 1093 bytes} [Mon Aug 29 21:21:00 2022] GET /identity/v3/users/nova => generated 82 bytes in 228 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:21:00.859181 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0b473aa-edd0-4017-83cc-b45a12b1a2e9 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:00.879176 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-175b5d8b-d797-4ea5-a916-e6666c1dd47a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:00.880348 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-175b5d8b-d797-4ea5-a916-e6666c1dd47a None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:00.880835 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-175b5d8b-d797-4ea5-a916-e6666c1dd47a None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:00.881358 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-175b5d8b-d797-4ea5-a916-e6666c1dd47a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:00.882390 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-175b5d8b-d797-4ea5-a916-e6666c1dd47a None admin] RBAC: Adding query filter params (role.id=a94bdb4c5796428baf63644c062dfcb2, scope.project.id=ec03e112e3d547fbab6faf7b485c3fca, user.id=139bbbde6d884b42b5440727926e2c58) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:00.883019 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-175b5d8b-d797-4ea5-a916-e6666c1dd47a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:00.884744 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-175b5d8b-d797-4ea5-a916-e6666c1dd47a None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:00.897528 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 348/697] 10.222.0.90 () {56 vars in 1374 bytes} [Mon Aug 29 21:21:00 2022] GET /identity/v3/role_assignments?user.id=139bbbde6d884b42b5440727926e2c58&scope.project.id=ec03e112e3d547fbab6faf7b485c3fca&role.id=a94bdb4c5796428baf63644c062dfcb2 => generated 608 bytes in 230 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:21:00.972426 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0b473aa-edd0-4017-83cc-b45a12b1a2e9 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:01.010837 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f19b0b4-0c61-45db-bf48-8aa44441c585 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:01.011507 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f19b0b4-0c61-45db-bf48-8aa44441c585 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:01.011792 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f19b0b4-0c61-45db-bf48-8aa44441c585 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:01.012066 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f19b0b4-0c61-45db-bf48-8aa44441c585 None admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:01.012859 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f19b0b4-0c61-45db-bf48-8aa44441c585 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:01.013173 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f19b0b4-0c61-45db-bf48-8aa44441c585 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:01.014788 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f19b0b4-0c61-45db-bf48-8aa44441c585 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:01.041839 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 350/698] 10.222.0.90 () {56 vars in 1133 bytes} [Mon Aug 29 21:21:00 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:01.043539 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-175b5d8b-d797-4ea5-a916-e6666c1dd47a None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:01.121052 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:01.166152 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-175b5d8b-d797-4ea5-a916-e6666c1dd47a None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:01.205586 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4557a324-0ee0-4037-8cf7-21c78b3c25b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:01.207100 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4557a324-0ee0-4037-8cf7-21c78b3c25b0 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:01.207100 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4557a324-0ee0-4037-8cf7-21c78b3c25b0 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:01.207620 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4557a324-0ee0-4037-8cf7-21c78b3c25b0 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:01.281021 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4557a324-0ee0-4037-8cf7-21c78b3c25b0 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:01.282565 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4557a324-0ee0-4037-8cf7-21c78b3c25b0 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:01.299127 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41a7f09c-363c-41a6-a442-ba336ede3237 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:01.299383 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41a7f09c-363c-41a6-a442-ba336ede3237 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:01.301070 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41a7f09c-363c-41a6-a442-ba336ede3237 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:01.305571 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 351/699] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:21:01 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 29 21:21:01.307151 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a5947e-8e49-4c0e-9c4e-8e952741cc3a None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:01.307432 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a5947e-8e49-4c0e-9c4e-8e952741cc3a None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:01.307706 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a5947e-8e49-4c0e-9c4e-8e952741cc3a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:01.326584 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-45a5947e-8e49-4c0e-9c4e-8e952741cc3a None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:21:01.327780 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-45a5947e-8e49-4c0e-9c4e-8e952741cc3a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:21:01.353135 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:01.358761 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:01.406233 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 349/700] 10.222.0.90 () {58 vars in 1282 bytes} [Mon Aug 29 21:21:01 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 363 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Aug 29 21:21:01.407565 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc33aa48-95a6-4c67-8a4b-a606a7f23ce3 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:01.407855 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc33aa48-95a6-4c67-8a4b-a606a7f23ce3 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:01.408134 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc33aa48-95a6-4c67-8a4b-a606a7f23ce3 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:01.410270 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 350/701] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:21:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:21:01.414525 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc33aa48-95a6-4c67-8a4b-a606a7f23ce3 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:01.415356 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 352/702] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:21:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 29 21:21:01.416703 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-378703c3-1318-4d33-9829-62759b330671 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:01.417000 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-378703c3-1318-4d33-9829-62759b330671 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:01.417277 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-378703c3-1318-4d33-9829-62759b330671 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:01.442510 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-378703c3-1318-4d33-9829-62759b330671 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:21:01.443729 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-378703c3-1318-4d33-9829-62759b330671 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:21:01.529336 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 353/703] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:21:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 21:21:01.547517 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc33aa48-95a6-4c67-8a4b-a606a7f23ce3 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:01.583773 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91d3c296-af98-4969-a911-3909c19592f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:01.584698 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d3c296-af98-4969-a911-3909c19592f7 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:01.585152 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d3c296-af98-4969-a911-3909c19592f7 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:01.585567 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d3c296-af98-4969-a911-3909c19592f7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:01.597510 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91d3c296-af98-4969-a911-3909c19592f7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:01.599284 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91d3c296-af98-4969-a911-3909c19592f7 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:01.614826 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-91d3c296-af98-4969-a911-3909c19592f7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:21:01.614826 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:01.614826 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:01.614826 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:01.614826 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:01.614826 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:01.614826 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:01.614826 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:01.614826 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:01.614826 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:01.614826 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:01.614826 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:01.614826 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 21:21:01.614826 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 21:21:01.614826 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 21:21:01.614826 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 21:21:01.615940 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:01.615940 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:01.615940 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:01.615940 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:01.615940 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:01.615940 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:01.615940 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:01.615940 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:01.615940 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:01.615940 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:01.615940 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:01.615940 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:01.615940 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:01.615940 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:01.615940 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:01.615940 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:01.615940 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 21:21:01.616883 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 21:21:01.616883 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:21:01.616883 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:21:01.622979 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 351/704] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:21:01 2022] GET /identity/v3/domains/Default => generated 87 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 29 21:21:01.624818 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-378703c3-1318-4d33-9829-62759b330671 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:01.724032 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:01.730706 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-378703c3-1318-4d33-9829-62759b330671 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:01.766285 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d17a466-c6b7-4263-96ee-d6c6fb1c3b8e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:01.766902 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d17a466-c6b7-4263-96ee-d6c6fb1c3b8e None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:01.767186 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d17a466-c6b7-4263-96ee-d6c6fb1c3b8e None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:01.767455 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d17a466-c6b7-4263-96ee-d6c6fb1c3b8e None admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:01.768662 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d17a466-c6b7-4263-96ee-d6c6fb1c3b8e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:01.768993 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d17a466-c6b7-4263-96ee-d6c6fb1c3b8e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:01.770567 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d17a466-c6b7-4263-96ee-d6c6fb1c3b8e None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:01.785592 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 354/705] 10.222.0.90 () {56 vars in 1109 bytes} [Mon Aug 29 21:21:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:01.788588 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91d3c296-af98-4969-a911-3909c19592f7 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:01.801230 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d17a466-c6b7-4263-96ee-d6c6fb1c3b8e None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:01.822146 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:01.945318 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91d3c296-af98-4969-a911-3909c19592f7 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:01.965145 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d17a466-c6b7-4263-96ee-d6c6fb1c3b8e None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:01.984059 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6711df0-7101-42df-9636-dc4c14334731 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:01.984675 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6711df0-7101-42df-9636-dc4c14334731 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:01.985055 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6711df0-7101-42df-9636-dc4c14334731 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:01.985449 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6711df0-7101-42df-9636-dc4c14334731 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:01.993285 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6711df0-7101-42df-9636-dc4c14334731 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:01.995089 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6711df0-7101-42df-9636-dc4c14334731 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:02.008211 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47f83a6e-7d68-4c55-9b94-dbb15214e939 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:02.008360 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-c6711df0-7101-42df-9636-dc4c14334731 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 21:21:02.008360 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:02.008360 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:02.008360 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:02.008360 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:02.008360 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:02.008360 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:02.008360 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:02.008360 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:02.008360 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:02.008360 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:02.008360 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:02.008360 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 29 21:21:02.008360 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 29 21:21:02.008360 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 29 21:21:02.009220 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 29 21:21:02.009220 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:02.009220 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:02.009220 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:02.009220 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:02.009220 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:02.009220 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:02.009220 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:02.009220 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:02.009220 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:02.009220 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:02.009220 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:02.009220 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:02.009220 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:02.009220 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:02.009220 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:02.009220 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:02.010166 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 29 21:21:02.010166 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 29 21:21:02.010166 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 29 21:21:02.010166 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 29 21:21:02.010166 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 29 21:21:02.010166 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 29 21:21:02.010166 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 21:21:02.010166 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:21:02.010618 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f83a6e-7d68-4c55-9b94-dbb15214e939 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:02.010618 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f83a6e-7d68-4c55-9b94-dbb15214e939 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:02.010618 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f83a6e-7d68-4c55-9b94-dbb15214e939 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:02.010816 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 352/706] 10.222.0.90 () {56 vars in 1111 bytes} [Mon Aug 29 21:21:01 2022] GET /identity/v3/projects/service => generated 88 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:21:02.010816 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6711df0-7101-42df-9636-dc4c14334731 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:02.014346 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47f83a6e-7d68-4c55-9b94-dbb15214e939 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:02.015716 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47f83a6e-7d68-4c55-9b94-dbb15214e939 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:02.029316 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-47f83a6e-7d68-4c55-9b94-dbb15214e939 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 29 21:21:02.029316 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:02.029316 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 29 21:21:02.029316 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 29 21:21:02.029316 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 29 21:21:02.029316 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 29 21:21:02.029316 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 29 21:21:02.029316 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 21:21:02.029316 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 29 21:21:02.029316 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:21:02.029316 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 29 21:21:02.029316 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:21:02.029316 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:02.029316 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:02.029316 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:02.029316 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:02.029316 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:02.030427 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:02.030427 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:02.030427 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:02.030427 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:02.030427 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:02.030427 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:02.030427 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 29 21:21:02.030427 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 29 21:21:02.030427 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 29 21:21:02.030427 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 29 21:21:02.030427 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:02.030427 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:02.030427 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:02.030427 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:02.030427 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:02.030427 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:02.031301 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:02.031301 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:02.031301 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:02.031301 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:02.031301 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:02.031301 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:02.031301 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:02.031301 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:02.031301 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:02.031301 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:02.031301 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 29 21:21:02.031301 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 21:21:02.031301 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 29 21:21:02.031301 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:21:02.031301 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 355/707] 10.222.0.90 () {56 vars in 1129 bytes} [Mon Aug 29 21:21:01 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 230 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:21:02.033154 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47f83a6e-7d68-4c55-9b94-dbb15214e939 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:02.051862 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:02.112418 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:02.145636 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47f83a6e-7d68-4c55-9b94-dbb15214e939 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:02.148780 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6711df0-7101-42df-9636-dc4c14334731 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:02.183265 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-178e215c-1c91-4b1a-9b7f-0465de4308c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:02.183975 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-178e215c-1c91-4b1a-9b7f-0465de4308c8 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:02.184328 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-178e215c-1c91-4b1a-9b7f-0465de4308c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:02.184674 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-178e215c-1c91-4b1a-9b7f-0465de4308c8 None admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:02.185444 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-178e215c-1c91-4b1a-9b7f-0465de4308c8 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:02.185880 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-178e215c-1c91-4b1a-9b7f-0465de4308c8 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:02.187444 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-178e215c-1c91-4b1a-9b7f-0465de4308c8 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:02.188957 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6510ae38-78cd-47f1-b729-48937c959604 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:02.189595 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6510ae38-78cd-47f1-b729-48937c959604 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:02.190572 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6510ae38-78cd-47f1-b729-48937c959604 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:02.190956 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6510ae38-78cd-47f1-b729-48937c959604 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:02.200952 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 356/708] 10.222.0.90 () {56 vars in 1148 bytes} [Mon Aug 29 21:21:02 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:02.201824 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-178e215c-1c91-4b1a-9b7f-0465de4308c8 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:02.206798 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6510ae38-78cd-47f1-b729-48937c959604 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:02.208256 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6510ae38-78cd-47f1-b729-48937c959604 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:02.223713 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-6510ae38-78cd-47f1-b729-48937c959604 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:21:02.223713 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:02.223713 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:02.223713 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:02.223713 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:02.223713 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:02.223713 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:02.223713 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:02.223713 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:02.223713 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:02.223713 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:02.223713 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:02.223713 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 21:21:02.223713 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 21:21:02.223713 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 21:21:02.223713 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 21:21:02.225997 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:02.225997 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:02.225997 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:02.225997 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:02.225997 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:02.225997 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:02.225997 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:02.225997 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:02.225997 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:02.225997 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:02.225997 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:02.225997 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:02.225997 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:02.225997 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:02.225997 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:02.225997 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:02.225997 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 21:21:02.228196 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 21:21:02.228196 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:21:02.228196 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:21:02.228196 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 353/709] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:21:02 2022] GET /identity/v3/domains/Default => generated 87 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:21:02.228196 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6510ae38-78cd-47f1-b729-48937c959604 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:02.294349 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:02.320466 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-178e215c-1c91-4b1a-9b7f-0465de4308c8 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:02.331897 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6510ae38-78cd-47f1-b729-48937c959604 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:02.357096 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93495cf7-71b1-441d-9438-a61a0c6da3f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:02.357712 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93495cf7-71b1-441d-9438-a61a0c6da3f6 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:02.358073 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93495cf7-71b1-441d-9438-a61a0c6da3f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:02.358358 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93495cf7-71b1-441d-9438-a61a0c6da3f6 None admin] PATH_INFO: `/v3/services` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:02.358995 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93495cf7-71b1-441d-9438-a61a0c6da3f6 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:02.359301 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93495cf7-71b1-441d-9438-a61a0c6da3f6 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:02.359488 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:02.360805 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93495cf7-71b1-441d-9438-a61a0c6da3f6 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:02.367599 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 357/710] 10.222.0.90 () {56 vars in 1124 bytes} [Mon Aug 29 21:21:02 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:02.368479 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93495cf7-71b1-441d-9438-a61a0c6da3f6 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:02.369429 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7cacc69-5d53-401e-ad8e-af0423c24fca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:02.370072 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7cacc69-5d53-401e-ad8e-af0423c24fca None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:02.370347 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7cacc69-5d53-401e-ad8e-af0423c24fca None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:02.370615 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7cacc69-5d53-401e-ad8e-af0423c24fca None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:02.371332 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7cacc69-5d53-401e-ad8e-af0423c24fca None admin] RBAC: Adding query filter params (role.id=a6e562cdc96c4be691d638990f8403fd, scope.project.id=ec03e112e3d547fbab6faf7b485c3fca, user.id=a35b5b7dea7b421da056f063d2436ffb) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:02.371712 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7cacc69-5d53-401e-ad8e-af0423c24fca None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:02.373133 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7cacc69-5d53-401e-ad8e-af0423c24fca None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:02.385940 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 354/711] 10.222.0.90 () {56 vars in 1374 bytes} [Mon Aug 29 21:21:02 2022] GET /identity/v3/role_assignments?user.id=a35b5b7dea7b421da056f063d2436ffb&scope.project.id=ec03e112e3d547fbab6faf7b485c3fca&role.id=a6e562cdc96c4be691d638990f8403fd => generated 259 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:02.386987 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7cacc69-5d53-401e-ad8e-af0423c24fca None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:02.493762 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93495cf7-71b1-441d-9438-a61a0c6da3f6 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:02.535269 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c173ab91-b9ca-43ae-ae5b-6ac90496cb37 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:02.536080 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c173ab91-b9ca-43ae-ae5b-6ac90496cb37 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:02.536248 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c173ab91-b9ca-43ae-ae5b-6ac90496cb37 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:02.536426 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c173ab91-b9ca-43ae-ae5b-6ac90496cb37 None admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:02.537111 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c173ab91-b9ca-43ae-ae5b-6ac90496cb37 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:02.537417 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c173ab91-b9ca-43ae-ae5b-6ac90496cb37 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:02.538871 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c173ab91-b9ca-43ae-ae5b-6ac90496cb37 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:02.567959 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 358/712] 10.222.0.90 () {56 vars in 1109 bytes} [Mon Aug 29 21:21:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:02.573806 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c173ab91-b9ca-43ae-ae5b-6ac90496cb37 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:02.578899 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:02.619767 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7cacc69-5d53-401e-ad8e-af0423c24fca None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:02.697445 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49138caf-cdcd-44dd-ae75-ef67a55c5519 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:02.698158 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49138caf-cdcd-44dd-ae75-ef67a55c5519 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:02.698557 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49138caf-cdcd-44dd-ae75-ef67a55c5519 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:02.698833 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49138caf-cdcd-44dd-ae75-ef67a55c5519 None admin] PATH_INFO: `/v3/services` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:02.699592 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49138caf-cdcd-44dd-ae75-ef67a55c5519 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:02.699946 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49138caf-cdcd-44dd-ae75-ef67a55c5519 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:02.701318 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49138caf-cdcd-44dd-ae75-ef67a55c5519 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:02.702037 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c173ab91-b9ca-43ae-ae5b-6ac90496cb37 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:02.708105 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 355/713] 10.222.0.90 () {56 vars in 1124 bytes} [Mon Aug 29 21:21:02 2022] GET /identity/v3/services?type=block-storage => generated 130 bytes in 321 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:02.767656 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-986ec03c-ddfe-476a-9e9d-752fe0befa31 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:02.768444 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-986ec03c-ddfe-476a-9e9d-752fe0befa31 None admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:02.768810 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-986ec03c-ddfe-476a-9e9d-752fe0befa31 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:02.769084 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-986ec03c-ddfe-476a-9e9d-752fe0befa31 None admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:02.770045 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-986ec03c-ddfe-476a-9e9d-752fe0befa31 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:02.771483 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-986ec03c-ddfe-476a-9e9d-752fe0befa31 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:02.780600 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28a9c89d-9142-4268-acf6-a6de798f5326 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:02.780889 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28a9c89d-9142-4268-acf6-a6de798f5326 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:02.781168 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28a9c89d-9142-4268-acf6-a6de798f5326 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:02.782848 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 356/714] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:21:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 29 21:21:02.790963 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57736a1d-4817-4b8d-b2fd-0158d1799f25 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:02.791261 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57736a1d-4817-4b8d-b2fd-0158d1799f25 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:02.791544 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57736a1d-4817-4b8d-b2fd-0158d1799f25 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:02.808181 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 359/715] 10.222.0.90 () {60 vars in 1132 bytes} [Mon Aug 29 21:21:02 2022] POST /identity/v3/users => generated 254 bytes in 235 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 21:21:02.847922 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-57736a1d-4817-4b8d-b2fd-0158d1799f25 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:21:02.849108 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-57736a1d-4817-4b8d-b2fd-0158d1799f25 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:21:03.005517 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:03.025232 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 357/716] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:21:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 235 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 29 21:21:03.410609 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-986ec03c-ddfe-476a-9e9d-752fe0befa31 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:03.485271 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:03.546189 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-986ec03c-ddfe-476a-9e9d-752fe0befa31 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:03.595409 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be57f935-e5ae-45ca-93f9-f759e7089611 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:03.596653 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be57f935-e5ae-45ca-93f9-f759e7089611 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:03.596936 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be57f935-e5ae-45ca-93f9-f759e7089611 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:03.597214 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be57f935-e5ae-45ca-93f9-f759e7089611 None admin] PATH_INFO: `/v3/services/network` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:03.597815 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be57f935-e5ae-45ca-93f9-f759e7089611 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:03.599209 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be57f935-e5ae-45ca-93f9-f759e7089611 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:03.613034 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-be57f935-e5ae-45ca-93f9-f759e7089611 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Aug 29 21:21:03.613034 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:03.613034 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 29 21:21:03.613034 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 29 21:21:03.613034 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 29 21:21:03.613034 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 29 21:21:03.613034 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 29 21:21:03.613034 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 21:21:03.613034 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Aug 29 21:21:03.613034 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:21:03.613034 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 29 21:21:03.613034 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:21:03.613034 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:03.613034 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:03.613034 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:03.613034 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:03.613034 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:03.614121 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:03.614121 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:03.614121 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:03.614121 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:03.614121 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:03.614121 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:03.614121 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 29 21:21:03.614121 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 29 21:21:03.614121 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 29 21:21:03.614121 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 29 21:21:03.614121 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:03.614121 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:03.614121 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:03.614121 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:03.614121 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:03.614121 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:03.614998 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:03.614998 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:03.614998 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:03.614998 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:03.614998 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:03.614998 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:03.614998 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:03.614998 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:03.614998 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:03.614998 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:03.614998 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 29 21:21:03.614998 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 21:21:03.614998 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Aug 29 21:21:03.614998 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:21:03.617246 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 360/717] 10.222.0.90 () {56 vars in 1111 bytes} [Mon Aug 29 21:21:03 2022] GET /identity/v3/services/network => generated 88 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:21:03.622877 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57736a1d-4817-4b8d-b2fd-0158d1799f25 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:03.682151 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b15a0fea-7924-4256-a879-02c1df70ab95 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:03.682570 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b15a0fea-7924-4256-a879-02c1df70ab95 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:03.682738 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b15a0fea-7924-4256-a879-02c1df70ab95 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:03.683977 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 361/718] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:21:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 29 21:21:03.689164 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d5d0da8-6c62-425f-b9d5-837c805a9aa0 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:03.689409 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d5d0da8-6c62-425f-b9d5-837c805a9aa0 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:03.689610 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d5d0da8-6c62-425f-b9d5-837c805a9aa0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:03.709909 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-3d5d0da8-6c62-425f-b9d5-837c805a9aa0 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:21:03.711221 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-3d5d0da8-6c62-425f-b9d5-837c805a9aa0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:21:03.788155 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 362/719] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:21:03 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 29 21:21:03.824123 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57736a1d-4817-4b8d-b2fd-0158d1799f25 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:03.888021 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a53d19c7-61a5-4f44-bf2d-587003ca3965 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:03.888646 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a53d19c7-61a5-4f44-bf2d-587003ca3965 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:03.888930 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a53d19c7-61a5-4f44-bf2d-587003ca3965 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:03.889205 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a53d19c7-61a5-4f44-bf2d-587003ca3965 None admin] PATH_INFO: `/v3/services` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:03.889869 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a53d19c7-61a5-4f44-bf2d-587003ca3965 None admin] RBAC: Adding query filter params (name=network) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:03.890175 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a53d19c7-61a5-4f44-bf2d-587003ca3965 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:03.891657 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a53d19c7-61a5-4f44-bf2d-587003ca3965 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:03.897276 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 358/720] 10.222.0.90 () {56 vars in 1112 bytes} [Mon Aug 29 21:21:03 2022] GET /identity/v3/services?name=network => generated 124 bytes in 275 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:03.902719 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d5d0da8-6c62-425f-b9d5-837c805a9aa0 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:03.992025 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:04.062482 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d5d0da8-6c62-425f-b9d5-837c805a9aa0 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:04.114380 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dfb6edb-bf9e-497e-bdd2-497b2bee61c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:04.115006 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dfb6edb-bf9e-497e-bdd2-497b2bee61c3 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:04.115291 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dfb6edb-bf9e-497e-bdd2-497b2bee61c3 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:04.115565 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dfb6edb-bf9e-497e-bdd2-497b2bee61c3 None admin] PATH_INFO: `/v3/services` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:04.116211 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dfb6edb-bf9e-497e-bdd2-497b2bee61c3 None admin] RBAC: Adding query filter params (type=network) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:04.116515 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dfb6edb-bf9e-497e-bdd2-497b2bee61c3 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:04.117915 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dfb6edb-bf9e-497e-bdd2-497b2bee61c3 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:04.127016 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 363/721] 10.222.0.90 () {56 vars in 1112 bytes} [Mon Aug 29 21:21:03 2022] GET /identity/v3/services?type=network => generated 124 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:04.154679 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a53d19c7-61a5-4f44-bf2d-587003ca3965 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:04.214773 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:04.290049 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a53d19c7-61a5-4f44-bf2d-587003ca3965 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:04.326163 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b1295be-6f9e-44b4-b54f-dc319f981518 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:04.326798 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b1295be-6f9e-44b4-b54f-dc319f981518 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:04.327076 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b1295be-6f9e-44b4-b54f-dc319f981518 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:04.327346 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b1295be-6f9e-44b4-b54f-dc319f981518 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:04.337716 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b1295be-6f9e-44b4-b54f-dc319f981518 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 21:21:04.342574 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-7b1295be-6f9e-44b4-b54f-dc319f981518 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 21:21:04.342574 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:04.342574 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:04.342574 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:04.342574 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:04.342574 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:04.342574 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:04.342574 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:04.342574 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:04.342574 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:04.342574 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:04.342574 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:04.342574 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 21:21:04.342574 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 21:21:04.342574 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 21:21:04.342574 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 21:21:04.343527 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 21:21:04.343527 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 21:21:04.343527 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 21:21:04.343527 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 21:21:04.343527 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:04.343527 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:04.343527 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:04.343527 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:04.343527 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:04.343527 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:04.343527 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:04.343527 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:04.343527 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:04.343527 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:04.343527 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:04.343527 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:04.344395 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:04.344395 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:04.344395 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:04.344395 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:04.344395 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 21:21:04.344395 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 21:21:04.344395 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 21:21:04.344395 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 21:21:04.344395 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 21:21:04.344395 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 21:21:04.344395 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 21:21:04.344395 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:21:04.344395 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 359/722] 10.222.0.90 () {56 vars in 1096 bytes} [Mon Aug 29 21:21:04 2022] GET /identity/v3/roles/admin => generated 83 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:21:04.352365 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dfb6edb-bf9e-497e-bdd2-497b2bee61c3 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:04.473269 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dfb6edb-bf9e-497e-bdd2-497b2bee61c3 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:04.491898 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:04.512801 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-654174fb-45fc-4f3e-ac52-b3c9947d4406 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:04.513824 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-654174fb-45fc-4f3e-ac52-b3c9947d4406 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:04.514398 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-654174fb-45fc-4f3e-ac52-b3c9947d4406 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:04.514872 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-654174fb-45fc-4f3e-ac52-b3c9947d4406 None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:04.515744 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-654174fb-45fc-4f3e-ac52-b3c9947d4406 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:04.516275 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-654174fb-45fc-4f3e-ac52-b3c9947d4406 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:04.517959 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-654174fb-45fc-4f3e-ac52-b3c9947d4406 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:04.527569 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 364/723] 10.222.0.90 () {56 vars in 1099 bytes} [Mon Aug 29 21:21:04 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:04.533456 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b1295be-6f9e-44b4-b54f-dc319f981518 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:04.669859 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b1295be-6f9e-44b4-b54f-dc319f981518 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:04.681322 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:04.706409 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb38fd54-44d5-4805-b073-9c3256bb249c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:04.707048 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb38fd54-44d5-4805-b073-9c3256bb249c None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:04.707326 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb38fd54-44d5-4805-b073-9c3256bb249c None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:04.707601 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb38fd54-44d5-4805-b073-9c3256bb249c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:04.718682 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb38fd54-44d5-4805-b073-9c3256bb249c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:04.720134 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb38fd54-44d5-4805-b073-9c3256bb249c None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:04.735567 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-eb38fd54-44d5-4805-b073-9c3256bb249c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:21:04.735567 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:04.735567 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:04.735567 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:04.735567 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:04.735567 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:04.735567 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:04.735567 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:04.735567 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:04.735567 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:04.735567 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:04.735567 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:04.735567 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 21:21:04.735567 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 21:21:04.735567 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 21:21:04.735567 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 21:21:04.737271 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:04.737271 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:04.737271 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:04.737271 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:04.737271 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:04.737271 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:04.737271 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:04.737271 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:04.737271 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:04.737271 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:04.737271 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:04.737271 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:04.737271 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:04.737271 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:04.737271 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:04.737271 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:04.737271 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 21:21:04.738399 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 21:21:04.738399 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:21:04.738399 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:21:04.738399 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 360/724] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:21:04 2022] GET /identity/v3/domains/Default => generated 87 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:21:04.748331 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-654174fb-45fc-4f3e-ac52-b3c9947d4406 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:04.748996 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72032a61-f031-420e-b55c-0ceb30260c77 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:04.749163 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72032a61-f031-420e-b55c-0ceb30260c77 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:04.749327 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72032a61-f031-420e-b55c-0ceb30260c77 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:04.750944 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 361/725] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:21:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:21:04.767392 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e74969f-6951-49ab-b1d8-5ffb831bf898 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:04.767687 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e74969f-6951-49ab-b1d8-5ffb831bf898 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:04.767916 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e74969f-6951-49ab-b1d8-5ffb831bf898 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:04.795630 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-7e74969f-6951-49ab-b1d8-5ffb831bf898 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:21:04.796837 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-7e74969f-6951-49ab-b1d8-5ffb831bf898 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:21:04.929008 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-654174fb-45fc-4f3e-ac52-b3c9947d4406 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:04.950495 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 362/726] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:21:04 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 21:21:04.976872 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8afa6934-365d-42aa-a6ee-293bce0292c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:04.977894 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8afa6934-365d-42aa-a6ee-293bce0292c4 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:04.978403 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8afa6934-365d-42aa-a6ee-293bce0292c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:04.978898 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8afa6934-365d-42aa-a6ee-293bce0292c4 None admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:04.979774 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8afa6934-365d-42aa-a6ee-293bce0292c4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:04.980315 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8afa6934-365d-42aa-a6ee-293bce0292c4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:04.981936 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8afa6934-365d-42aa-a6ee-293bce0292c4 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:05.007389 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 365/727] 10.222.0.90 () {56 vars in 1109 bytes} [Mon Aug 29 21:21:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 259 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:05.012525 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e74969f-6951-49ab-b1d8-5ffb831bf898 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:05.173247 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:05.189573 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e74969f-6951-49ab-b1d8-5ffb831bf898 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:05.243214 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b2851ba-abc8-4876-bda6-178e2c2a890a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:05.244348 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b2851ba-abc8-4876-bda6-178e2c2a890a None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:05.244867 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b2851ba-abc8-4876-bda6-178e2c2a890a None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:05.245361 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b2851ba-abc8-4876-bda6-178e2c2a890a None admin] PATH_INFO: `/v3/users/swift` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:05.256505 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8afa6934-365d-42aa-a6ee-293bce0292c4 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:05.260934 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b2851ba-abc8-4876-bda6-178e2c2a890a None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:05.262445 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b2851ba-abc8-4876-bda6-178e2c2a890a None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:05.281824 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-5b2851ba-abc8-4876-bda6-178e2c2a890a None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Aug 29 21:21:05.281824 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:05.281824 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:05.281824 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:05.281824 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:05.281824 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:05.281824 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:05.281824 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:05.281824 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:05.281824 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:05.281824 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:05.281824 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:05.281824 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 29 21:21:05.281824 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 29 21:21:05.281824 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 29 21:21:05.281824 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 29 21:21:05.282865 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:05.282865 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:05.282865 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 29 21:21:05.282865 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 21:21:05.282865 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 29 21:21:05.282865 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 21:21:05.282865 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:05.282865 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:05.282865 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:05.282865 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:05.282865 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:05.282865 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:05.282865 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:05.282865 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:05.282865 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:05.282865 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:05.282865 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:05.282865 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:05.284592 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:05.284592 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:05.284592 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 29 21:21:05.284592 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 29 21:21:05.284592 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 29 21:21:05.284592 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 29 21:21:05.284592 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 29 21:21:05.284592 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 29 21:21:05.284592 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Aug 29 21:21:05.284592 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:21:05.284592 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 363/728] 10.222.0.90 () {56 vars in 1096 bytes} [Mon Aug 29 21:21:05 2022] GET /identity/v3/users/swift => generated 83 bytes in 268 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:21:05.287108 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b2851ba-abc8-4876-bda6-178e2c2a890a None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:05.310437 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:05.407839 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8afa6934-365d-42aa-a6ee-293bce0292c4 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:05.449492 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a95ed6d4-a762-49eb-bcc8-5466c0c25e55 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:05.450737 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a95ed6d4-a762-49eb-bcc8-5466c0c25e55 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:05.451682 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a95ed6d4-a762-49eb-bcc8-5466c0c25e55 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:05.452417 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a95ed6d4-a762-49eb-bcc8-5466c0c25e55 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:05.457982 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b2851ba-abc8-4876-bda6-178e2c2a890a None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:05.468055 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a95ed6d4-a762-49eb-bcc8-5466c0c25e55 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 21:21:05.475749 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-a95ed6d4-a762-49eb-bcc8-5466c0c25e55 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 21:21:05.475749 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:05.475749 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:05.475749 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:05.475749 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:05.475749 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:05.475749 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:05.475749 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:05.475749 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:05.475749 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:05.475749 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:05.475749 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:05.475749 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 21:21:05.475749 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 21:21:05.475749 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 21:21:05.475749 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 21:21:05.476820 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 21:21:05.476820 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 21:21:05.476820 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 21:21:05.476820 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 21:21:05.476820 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:05.476820 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:05.476820 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:05.476820 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:05.476820 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:05.476820 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:05.476820 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:05.476820 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:05.476820 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:05.476820 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:05.476820 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:05.476820 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:05.477691 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:05.477691 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:05.477691 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:05.477691 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:05.477691 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 21:21:05.477691 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 21:21:05.477691 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 21:21:05.477691 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 21:21:05.477691 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 21:21:05.477691 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 21:21:05.477691 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 21:21:05.477691 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:21:05.477691 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 366/729] 10.222.0.90 () {56 vars in 1096 bytes} [Mon Aug 29 21:21:05 2022] GET /identity/v3/roles/admin => generated 83 bytes in 222 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:21:05.479621 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52c70cb8-3650-45db-8322-c932d9586ba3 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:05.479982 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52c70cb8-3650-45db-8322-c932d9586ba3 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:05.480325 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52c70cb8-3650-45db-8322-c932d9586ba3 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:05.482416 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 367/730] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:21:05 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:21:05.486553 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52c70cb8-3650-45db-8322-c932d9586ba3 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:05.503558 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea772f87-1a32-4a7d-b181-0f2ab060eea2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:05.504511 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea772f87-1a32-4a7d-b181-0f2ab060eea2 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:05.504979 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea772f87-1a32-4a7d-b181-0f2ab060eea2 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:05.505451 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea772f87-1a32-4a7d-b181-0f2ab060eea2 None admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:05.506502 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea772f87-1a32-4a7d-b181-0f2ab060eea2 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:05.507113 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea772f87-1a32-4a7d-b181-0f2ab060eea2 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:05.509050 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea772f87-1a32-4a7d-b181-0f2ab060eea2 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:05.534004 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 364/731] 10.222.0.90 () {56 vars in 1135 bytes} [Mon Aug 29 21:21:05 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 245 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:05.537913 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a318f1c-e877-4334-8874-f786e23dacf7 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:05.538320 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a318f1c-e877-4334-8874-f786e23dacf7 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:05.538677 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a318f1c-e877-4334-8874-f786e23dacf7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:05.558140 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-9a318f1c-e877-4334-8874-f786e23dacf7 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:21:05.559505 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-9a318f1c-e877-4334-8874-f786e23dacf7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:21:05.563696 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:05.619118 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52c70cb8-3650-45db-8322-c932d9586ba3 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:05.636847 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:05.653810 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 365/732] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:21:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 21:21:05.653810 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a318f1c-e877-4334-8874-f786e23dacf7 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:05.671070 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fe73521-6f1a-4db3-ade2-9db6182eacfd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:05.671687 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fe73521-6f1a-4db3-ade2-9db6182eacfd None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:05.671965 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fe73521-6f1a-4db3-ade2-9db6182eacfd None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:05.672234 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fe73521-6f1a-4db3-ade2-9db6182eacfd None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:05.672905 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fe73521-6f1a-4db3-ade2-9db6182eacfd None admin] RBAC: Adding query filter params (name=admin) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:05.673203 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fe73521-6f1a-4db3-ade2-9db6182eacfd None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:05.674967 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fe73521-6f1a-4db3-ade2-9db6182eacfd None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:05.684012 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 368/733] 10.222.0.90 () {56 vars in 1099 bytes} [Mon Aug 29 21:21:05 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:05.685209 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-363c1c31-c6bb-4e8c-9c08-4f1580bbb49f None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:05.685491 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-363c1c31-c6bb-4e8c-9c08-4f1580bbb49f None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:05.685804 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-363c1c31-c6bb-4e8c-9c08-4f1580bbb49f None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:05.687385 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 369/734] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:21:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:21:05.688930 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-363c1c31-c6bb-4e8c-9c08-4f1580bbb49f None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:05.761797 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a318f1c-e877-4334-8874-f786e23dacf7 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:05.804731 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57392c75-4f8e-4d0b-9f15-01a1b85cfbfb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:05.805346 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57392c75-4f8e-4d0b-9f15-01a1b85cfbfb None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:05.805816 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57392c75-4f8e-4d0b-9f15-01a1b85cfbfb None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:05.806736 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57392c75-4f8e-4d0b-9f15-01a1b85cfbfb None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:05.820085 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-363c1c31-c6bb-4e8c-9c08-4f1580bbb49f None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:05.821605 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:05.872134 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a743473-ab29-4c87-9858-2f835f84739b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:05.872996 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a743473-ab29-4c87-9858-2f835f84739b None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:05.873459 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a743473-ab29-4c87-9858-2f835f84739b None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:05.873984 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a743473-ab29-4c87-9858-2f835f84739b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:05.886615 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a743473-ab29-4c87-9858-2f835f84739b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:05.888379 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a743473-ab29-4c87-9858-2f835f84739b None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:05.899764 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57392c75-4f8e-4d0b-9f15-01a1b85cfbfb None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:05.901217 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57392c75-4f8e-4d0b-9f15-01a1b85cfbfb None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:05.904997 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-3a743473-ab29-4c87-9858-2f835f84739b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:21:05.904997 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:05.904997 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:05.904997 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:05.904997 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:05.904997 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:05.904997 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:05.904997 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:05.904997 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:05.904997 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:05.904997 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:05.904997 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:05.904997 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 21:21:05.904997 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 21:21:05.904997 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 21:21:05.904997 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 21:21:05.906147 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:05.906147 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:05.906147 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:05.906147 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:05.906147 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:05.906147 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:05.906147 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:05.906147 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:05.906147 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:05.906147 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:05.906147 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:05.906147 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:05.906147 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:05.906147 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:05.906147 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:05.906147 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:05.906147 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 21:21:05.907145 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 21:21:05.907145 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:21:05.907145 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:21:05.909155 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 370/735] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:21:05 2022] GET /identity/v3/domains/Default => generated 87 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:21:05.910724 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be33059e-7f3c-4fcf-921d-f2422d7f4b61 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:05.911209 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be33059e-7f3c-4fcf-921d-f2422d7f4b61 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:05.911648 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be33059e-7f3c-4fcf-921d-f2422d7f4b61 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:05.932139 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-be33059e-7f3c-4fcf-921d-f2422d7f4b61 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:21:05.933354 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-be33059e-7f3c-4fcf-921d-f2422d7f4b61 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:21:05.941832 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:06.005947 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 371/736] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:21:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 21:21:06.007055 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be33059e-7f3c-4fcf-921d-f2422d7f4b61 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:06.024621 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 366/737] 10.222.0.90 () {58 vars in 1282 bytes} [Mon Aug 29 21:21:05 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 373 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 29 21:21:06.025820 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57392c75-4f8e-4d0b-9f15-01a1b85cfbfb None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:06.085745 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:06.124308 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be33059e-7f3c-4fcf-921d-f2422d7f4b61 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:06.162303 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-847e3675-ad60-4ac8-9ba2-98bdc4f26f74 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:06.163298 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-847e3675-ad60-4ac8-9ba2-98bdc4f26f74 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:06.163782 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-847e3675-ad60-4ac8-9ba2-98bdc4f26f74 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:06.164256 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-847e3675-ad60-4ac8-9ba2-98bdc4f26f74 None admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:06.164639 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57392c75-4f8e-4d0b-9f15-01a1b85cfbfb None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:06.165372 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-847e3675-ad60-4ac8-9ba2-98bdc4f26f74 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:06.165920 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-847e3675-ad60-4ac8-9ba2-98bdc4f26f74 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:06.167550 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-847e3675-ad60-4ac8-9ba2-98bdc4f26f74 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:06.181026 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 372/738] 10.222.0.90 () {56 vars in 1109 bytes} [Mon Aug 29 21:21:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:06.182200 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-847e3675-ad60-4ac8-9ba2-98bdc4f26f74 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:06.185477 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:06.209482 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9843a216-62f2-4a4c-ac9f-28d0e4bb7431 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:06.210263 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9843a216-62f2-4a4c-ac9f-28d0e4bb7431 None admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:06.210547 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9843a216-62f2-4a4c-ac9f-28d0e4bb7431 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:06.210819 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9843a216-62f2-4a4c-ac9f-28d0e4bb7431 None admin] PATH_INFO: `/v3/services` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:06.211471 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9843a216-62f2-4a4c-ac9f-28d0e4bb7431 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:06.213530 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9843a216-62f2-4a4c-ac9f-28d0e4bb7431 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:06.247625 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 367/739] 10.222.0.90 () {60 vars in 1141 bytes} [Mon Aug 29 21:21:06 2022] POST /identity/v3/services => generated 251 bytes in 220 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:21:06.247625 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9843a216-62f2-4a4c-ac9f-28d0e4bb7431 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:06.338999 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-847e3675-ad60-4ac8-9ba2-98bdc4f26f74 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:06.380234 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:06.384799 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b25a003d-ca06-49aa-bf0f-abdc99e5a6dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:06.385446 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b25a003d-ca06-49aa-bf0f-abdc99e5a6dc None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:06.385731 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b25a003d-ca06-49aa-bf0f-abdc99e5a6dc None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:06.386029 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b25a003d-ca06-49aa-bf0f-abdc99e5a6dc None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:06.386729 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9843a216-62f2-4a4c-ac9f-28d0e4bb7431 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:06.399492 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b25a003d-ca06-49aa-bf0f-abdc99e5a6dc None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:06.401062 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b25a003d-ca06-49aa-bf0f-abdc99e5a6dc None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:06.416672 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-b25a003d-ca06-49aa-bf0f-abdc99e5a6dc None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:21:06.416672 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:06.416672 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:06.416672 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:06.416672 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:06.416672 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:06.416672 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:06.416672 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:06.416672 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:06.416672 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:06.416672 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:06.416672 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:06.416672 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 21:21:06.416672 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 21:21:06.416672 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 21:21:06.416672 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 21:21:06.417718 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:06.417718 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:06.417718 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:06.417718 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:06.417718 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:06.417718 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:06.417718 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:06.417718 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:06.417718 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:06.417718 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:06.417718 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:06.417718 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:06.417718 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:06.417718 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:06.417718 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:06.417718 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:06.417718 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 21:21:06.419972 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 21:21:06.419972 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:21:06.419972 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:21:06.419972 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 373/740] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:21:06 2022] GET /identity/v3/domains/Default => generated 87 bytes in 236 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:21:06.419972 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b25a003d-ca06-49aa-bf0f-abdc99e5a6dc None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:06.438651 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1628d3c2-ec36-489e-98cb-2d05546b0943 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:06.440201 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1628d3c2-ec36-489e-98cb-2d05546b0943 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:06.440794 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1628d3c2-ec36-489e-98cb-2d05546b0943 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:06.441390 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1628d3c2-ec36-489e-98cb-2d05546b0943 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:06.450203 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1628d3c2-ec36-489e-98cb-2d05546b0943 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:06.451894 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1628d3c2-ec36-489e-98cb-2d05546b0943 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:06.464932 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-1628d3c2-ec36-489e-98cb-2d05546b0943 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 29 21:21:06.464932 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:06.464932 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:06.464932 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:06.464932 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:06.464932 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:06.464932 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:06.464932 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:06.464932 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:06.464932 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:06.464932 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:06.464932 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:06.464932 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 29 21:21:06.464932 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 29 21:21:06.464932 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 29 21:21:06.464932 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 29 21:21:06.466214 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:06.466214 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:06.466214 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 29 21:21:06.466214 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 21:21:06.466214 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 29 21:21:06.466214 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 21:21:06.466214 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:06.466214 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:06.466214 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:06.466214 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:06.466214 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:06.466214 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:06.466214 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:06.466214 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:06.466214 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:06.466214 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:06.466214 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:06.466214 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:06.467203 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:06.467203 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:06.467203 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 29 21:21:06.467203 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 29 21:21:06.467203 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 29 21:21:06.467203 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 29 21:21:06.467203 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 29 21:21:06.467203 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 29 21:21:06.467203 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 29 21:21:06.467203 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:21:06.467203 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 368/741] 10.222.0.90 () {56 vars in 1093 bytes} [Mon Aug 29 21:21:06 2022] GET /identity/v3/users/nova => generated 82 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:21:06.467810 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1628d3c2-ec36-489e-98cb-2d05546b0943 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:06.480937 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:06.537068 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b25a003d-ca06-49aa-bf0f-abdc99e5a6dc None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:06.572701 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7ae7c28-b2e0-4e61-a6a3-e34a3eb75db2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:06.573604 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ae7c28-b2e0-4e61-a6a3-e34a3eb75db2 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:06.573911 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ae7c28-b2e0-4e61-a6a3-e34a3eb75db2 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:06.574185 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ae7c28-b2e0-4e61-a6a3-e34a3eb75db2 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:06.585259 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7ae7c28-b2e0-4e61-a6a3-e34a3eb75db2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 21:21:06.589318 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1628d3c2-ec36-489e-98cb-2d05546b0943 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:06.591008 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-f7ae7c28-b2e0-4e61-a6a3-e34a3eb75db2 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 29 21:21:06.591008 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:06.591008 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:06.591008 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:06.591008 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:06.591008 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:06.591008 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:06.591008 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:06.591008 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:06.591008 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:06.591008 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:06.591008 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:06.591008 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 21:21:06.591008 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 21:21:06.591008 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 21:21:06.591008 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 21:21:06.591969 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 21:21:06.591969 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 21:21:06.591969 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 21:21:06.591969 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 21:21:06.591969 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:06.591969 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:06.591969 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:06.591969 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:06.591969 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:06.591969 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:06.591969 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:06.591969 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:06.591969 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:06.591969 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:06.591969 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:06.591969 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:06.592855 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:06.592855 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:06.592855 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:06.592855 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:06.592855 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 21:21:06.592855 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 21:21:06.592855 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 21:21:06.592855 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 21:21:06.592855 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 21:21:06.592855 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 21:21:06.592855 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 29 21:21:06.592855 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:21:06.592855 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 374/742] 10.222.0.90 () {56 vars in 1102 bytes} [Mon Aug 29 21:21:06 2022] GET /identity/v3/roles/service => generated 85 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:21:06.592855 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7ae7c28-b2e0-4e61-a6a3-e34a3eb75db2 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:06.633618 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e39540e5-1406-4543-b490-5baf824924ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:06.634278 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e39540e5-1406-4543-b490-5baf824924ee None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:06.634558 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e39540e5-1406-4543-b490-5baf824924ee None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:06.634832 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e39540e5-1406-4543-b490-5baf824924ee None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:06.635473 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e39540e5-1406-4543-b490-5baf824924ee None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:06.635773 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e39540e5-1406-4543-b490-5baf824924ee None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:06.637151 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e39540e5-1406-4543-b490-5baf824924ee None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:06.653613 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 369/743] 10.222.0.90 () {56 vars in 1109 bytes} [Mon Aug 29 21:21:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:06.653613 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad265ad-fbf0-4e51-9cb4-21ddc9b44580 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:06.653613 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad265ad-fbf0-4e51-9cb4-21ddc9b44580 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:06.653613 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad265ad-fbf0-4e51-9cb4-21ddc9b44580 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:06.653613 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 370/744] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:21:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:21:06.653613 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dad265ad-fbf0-4e51-9cb4-21ddc9b44580 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:06.678649 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:06.716409 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7ae7c28-b2e0-4e61-a6a3-e34a3eb75db2 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:06.728908 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:06.755434 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feedec7d-a43c-4c3e-9a2d-b3464a1f827d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:06.756298 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feedec7d-a43c-4c3e-9a2d-b3464a1f827d None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:06.756587 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feedec7d-a43c-4c3e-9a2d-b3464a1f827d None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:06.756861 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feedec7d-a43c-4c3e-9a2d-b3464a1f827d None admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:06.757605 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feedec7d-a43c-4c3e-9a2d-b3464a1f827d None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:06.757943 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feedec7d-a43c-4c3e-9a2d-b3464a1f827d None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:06.759375 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feedec7d-a43c-4c3e-9a2d-b3464a1f827d None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:06.771262 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dad265ad-fbf0-4e51-9cb4-21ddc9b44580 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:06.780435 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 375/745] 10.222.0.90 () {56 vars in 1133 bytes} [Mon Aug 29 21:21:06 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:06.780435 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feedec7d-a43c-4c3e-9a2d-b3464a1f827d None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:06.809257 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45d48017-22d0-412e-a517-fdfbb3f57ac5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:06.809907 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45d48017-22d0-412e-a517-fdfbb3f57ac5 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:06.810190 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45d48017-22d0-412e-a517-fdfbb3f57ac5 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:06.810462 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45d48017-22d0-412e-a517-fdfbb3f57ac5 None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:06.811136 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45d48017-22d0-412e-a517-fdfbb3f57ac5 None admin] RBAC: Adding query filter params (name=service) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:06.811440 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45d48017-22d0-412e-a517-fdfbb3f57ac5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:06.812796 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45d48017-22d0-412e-a517-fdfbb3f57ac5 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:06.822426 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 371/746] 10.222.0.90 () {56 vars in 1103 bytes} [Mon Aug 29 21:21:06 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:06.823639 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18c4c34a-bc66-4fc8-934a-a79e4be31712 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:06.823925 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18c4c34a-bc66-4fc8-934a-a79e4be31712 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:06.824197 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18c4c34a-bc66-4fc8-934a-a79e4be31712 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:06.844930 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-18c4c34a-bc66-4fc8-934a-a79e4be31712 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:21:06.846744 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-18c4c34a-bc66-4fc8-934a-a79e4be31712 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:21:06.898514 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feedec7d-a43c-4c3e-9a2d-b3464a1f827d None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:06.926733 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 372/747] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:21:06 2022] POST /identity/v3/auth/tokens => generated 903 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 21:21:06.927470 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18c4c34a-bc66-4fc8-934a-a79e4be31712 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:06.942310 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:06.961854 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd324bfe-1f28-4efe-82ec-775dc34b294f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:06.961854 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd324bfe-1f28-4efe-82ec-775dc34b294f None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:06.961854 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd324bfe-1f28-4efe-82ec-775dc34b294f None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:06.961854 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd324bfe-1f28-4efe-82ec-775dc34b294f None admin] PATH_INFO: `/v3/projects/service` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:06.968243 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd324bfe-1f28-4efe-82ec-775dc34b294f None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:06.969756 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd324bfe-1f28-4efe-82ec-775dc34b294f None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:06.982789 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-bd324bfe-1f28-4efe-82ec-775dc34b294f None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 21:21:06.982789 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:06.982789 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:06.982789 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:06.982789 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:06.982789 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:06.982789 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:06.982789 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:06.982789 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:06.982789 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:06.982789 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:06.982789 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:06.982789 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 29 21:21:06.982789 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 29 21:21:06.982789 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 29 21:21:06.983726 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 29 21:21:06.983726 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:06.983726 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:06.983726 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:06.983726 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:06.983726 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:06.983726 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:06.983726 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:06.983726 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:06.983726 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:06.983726 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:06.983726 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:06.983726 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:06.983726 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:06.983726 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:06.983726 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:06.983726 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:06.985231 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 29 21:21:06.985231 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 29 21:21:06.985231 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 29 21:21:06.985231 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 29 21:21:06.985231 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 29 21:21:06.985231 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 29 21:21:06.985231 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 21:21:06.985231 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:21:06.985231 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 376/748] 10.222.0.90 () {56 vars in 1111 bytes} [Mon Aug 29 21:21:06 2022] GET /identity/v3/projects/service => generated 88 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:21:06.985231 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd324bfe-1f28-4efe-82ec-775dc34b294f None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:07.037332 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:07.047366 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18c4c34a-bc66-4fc8-934a-a79e4be31712 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:07.100568 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15cb5965-be5e-4a91-a560-6e4f4930beb6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:07.101142 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15cb5965-be5e-4a91-a560-6e4f4930beb6 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:07.101426 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15cb5965-be5e-4a91-a560-6e4f4930beb6 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:07.101702 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15cb5965-be5e-4a91-a560-6e4f4930beb6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:07.112631 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15cb5965-be5e-4a91-a560-6e4f4930beb6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:07.114115 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15cb5965-be5e-4a91-a560-6e4f4930beb6 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:07.119716 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd324bfe-1f28-4efe-82ec-775dc34b294f None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:07.129527 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-15cb5965-be5e-4a91-a560-6e4f4930beb6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:21:07.129527 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:07.129527 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:07.129527 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:07.129527 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:07.129527 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:07.129527 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:07.129527 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:07.129527 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:07.129527 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:07.129527 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:07.129527 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:07.129527 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 21:21:07.129527 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 21:21:07.129527 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 21:21:07.129527 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 21:21:07.130563 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:07.130563 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:07.130563 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:07.130563 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:07.130563 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:07.130563 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:07.130563 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:07.130563 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:07.130563 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:07.130563 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:07.130563 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:07.130563 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:07.130563 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:07.130563 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:07.130563 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:07.130563 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:07.130563 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 21:21:07.132034 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 21:21:07.132034 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:21:07.132034 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:21:07.132034 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 373/749] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:21:06 2022] GET /identity/v3/domains/Default => generated 87 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:21:07.132034 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15cb5965-be5e-4a91-a560-6e4f4930beb6 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:07.162292 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bdfcaa9-14f9-407b-8349-7cefad99dbe9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:07.162980 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bdfcaa9-14f9-407b-8349-7cefad99dbe9 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:07.163260 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bdfcaa9-14f9-407b-8349-7cefad99dbe9 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:07.163537 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bdfcaa9-14f9-407b-8349-7cefad99dbe9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:07.174048 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bdfcaa9-14f9-407b-8349-7cefad99dbe9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:07.176103 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bdfcaa9-14f9-407b-8349-7cefad99dbe9 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:07.192675 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-6bdfcaa9-14f9-407b-8349-7cefad99dbe9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:21:07.192675 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:07.192675 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:07.192675 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:07.192675 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:07.192675 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:07.192675 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:07.192675 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:07.192675 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:07.192675 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:07.192675 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:07.192675 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:07.192675 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 21:21:07.192675 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 21:21:07.192675 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 21:21:07.192675 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 21:21:07.193705 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:07.193705 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:07.193705 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:07.193705 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:07.193705 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:07.193705 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:07.193705 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:07.193705 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:07.193705 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:07.193705 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:07.193705 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:07.193705 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:07.193705 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:07.193705 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:07.193705 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:07.193705 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:07.193705 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 21:21:07.195841 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 21:21:07.195841 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:21:07.195841 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:21:07.195841 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 377/750] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:21:06 2022] GET /identity/v3/domains/Default => generated 87 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:21:07.195841 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bdfcaa9-14f9-407b-8349-7cefad99dbe9 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:07.199176 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:07.242418 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15cb5965-be5e-4a91-a560-6e4f4930beb6 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:07.284053 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-122b832f-1074-4bfc-8e1f-2c08d883d9cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:07.284677 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-122b832f-1074-4bfc-8e1f-2c08d883d9cb None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:07.284957 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-122b832f-1074-4bfc-8e1f-2c08d883d9cb None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:07.285414 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-122b832f-1074-4bfc-8e1f-2c08d883d9cb None admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:07.285977 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-122b832f-1074-4bfc-8e1f-2c08d883d9cb None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:07.286283 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-122b832f-1074-4bfc-8e1f-2c08d883d9cb None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:07.287706 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-122b832f-1074-4bfc-8e1f-2c08d883d9cb None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:07.288446 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:07.306919 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 374/751] 10.222.0.90 () {56 vars in 1148 bytes} [Mon Aug 29 21:21:07 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:07.307810 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-122b832f-1074-4bfc-8e1f-2c08d883d9cb None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:07.312099 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bdfcaa9-14f9-407b-8349-7cefad99dbe9 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:07.349702 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ef7e936-7f7d-4553-9fd3-bfcfbff3f07d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:07.350350 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ef7e936-7f7d-4553-9fd3-bfcfbff3f07d None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:07.350691 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ef7e936-7f7d-4553-9fd3-bfcfbff3f07d None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:07.350914 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ef7e936-7f7d-4553-9fd3-bfcfbff3f07d None admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:07.351577 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ef7e936-7f7d-4553-9fd3-bfcfbff3f07d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:07.351888 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ef7e936-7f7d-4553-9fd3-bfcfbff3f07d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:07.353377 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ef7e936-7f7d-4553-9fd3-bfcfbff3f07d None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:07.371206 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 378/752] 10.222.0.90 () {56 vars in 1109 bytes} [Mon Aug 29 21:21:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:07.371206 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ef7e936-7f7d-4553-9fd3-bfcfbff3f07d None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:07.418795 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-122b832f-1074-4bfc-8e1f-2c08d883d9cb None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:07.449318 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:07.460950 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02c4b494-5185-4e63-a3b1-46da2e4a4ff9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:07.461571 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c4b494-5185-4e63-a3b1-46da2e4a4ff9 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:07.461880 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c4b494-5185-4e63-a3b1-46da2e4a4ff9 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:07.462163 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c4b494-5185-4e63-a3b1-46da2e4a4ff9 None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:07.462811 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02c4b494-5185-4e63-a3b1-46da2e4a4ff9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:07.463113 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02c4b494-5185-4e63-a3b1-46da2e4a4ff9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:07.464475 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02c4b494-5185-4e63-a3b1-46da2e4a4ff9 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:07.481294 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 375/753] 10.222.0.90 () {56 vars in 1109 bytes} [Mon Aug 29 21:21:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:07.481294 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02c4b494-5185-4e63-a3b1-46da2e4a4ff9 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:07.481560 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ef7e936-7f7d-4553-9fd3-bfcfbff3f07d None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:07.516697 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c577cb64-ebb2-4ca5-9756-8481fcf43efa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:07.517313 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c577cb64-ebb2-4ca5-9756-8481fcf43efa None admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:07.517601 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c577cb64-ebb2-4ca5-9756-8481fcf43efa None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:07.518597 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c577cb64-ebb2-4ca5-9756-8481fcf43efa None admin] PATH_INFO: `/v3/services` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:07.519323 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c577cb64-ebb2-4ca5-9756-8481fcf43efa None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:07.520718 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c577cb64-ebb2-4ca5-9756-8481fcf43efa None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:07.532160 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:07.541838 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 379/754] 10.222.0.90 () {60 vars in 1141 bytes} [Mon Aug 29 21:21:07 2022] POST /identity/v3/services => generated 240 bytes in 172 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:21:07.541838 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c577cb64-ebb2-4ca5-9756-8481fcf43efa None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:07.603416 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02c4b494-5185-4e63-a3b1-46da2e4a4ff9 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:07.659446 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be5108e5-3081-4b7a-b0d6-6c4607a4196c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:07.660521 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be5108e5-3081-4b7a-b0d6-6c4607a4196c None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:07.660959 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be5108e5-3081-4b7a-b0d6-6c4607a4196c None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:07.661153 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c577cb64-ebb2-4ca5-9756-8481fcf43efa None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:07.661576 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be5108e5-3081-4b7a-b0d6-6c4607a4196c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:07.662496 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be5108e5-3081-4b7a-b0d6-6c4607a4196c None admin] RBAC: Adding query filter params (role.id=a6e562cdc96c4be691d638990f8403fd, scope.project.id=ec03e112e3d547fbab6faf7b485c3fca, user.id=139bbbde6d884b42b5440727926e2c58) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:07.663051 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be5108e5-3081-4b7a-b0d6-6c4607a4196c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:07.664857 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be5108e5-3081-4b7a-b0d6-6c4607a4196c None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:07.672001 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 376/755] 10.222.0.90 () {56 vars in 1374 bytes} [Mon Aug 29 21:21:07 2022] GET /identity/v3/role_assignments?user.id=139bbbde6d884b42b5440727926e2c58&scope.project.id=ec03e112e3d547fbab6faf7b485c3fca&role.id=a6e562cdc96c4be691d638990f8403fd => generated 259 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:07.673116 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be5108e5-3081-4b7a-b0d6-6c4607a4196c None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:07.721341 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05a80724-773a-4990-96bd-4ca6edb03842 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:07.721341 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05a80724-773a-4990-96bd-4ca6edb03842 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:07.722319 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05a80724-773a-4990-96bd-4ca6edb03842 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:07.723049 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05a80724-773a-4990-96bd-4ca6edb03842 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:07.733173 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05a80724-773a-4990-96bd-4ca6edb03842 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:07.735377 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05a80724-773a-4990-96bd-4ca6edb03842 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:07.738288 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:07.754617 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-05a80724-773a-4990-96bd-4ca6edb03842 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 21:21:07.754617 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:07.754617 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:07.754617 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:07.754617 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:07.754617 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:07.754617 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:07.754617 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:07.754617 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:07.754617 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:07.754617 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:07.754617 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:07.754617 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 29 21:21:07.754617 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 29 21:21:07.754617 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 29 21:21:07.755911 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 29 21:21:07.755911 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:07.755911 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:07.755911 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:07.755911 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:07.755911 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:07.755911 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:07.755911 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:07.755911 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:07.755911 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:07.755911 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:07.755911 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:07.755911 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:07.755911 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:07.755911 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:07.755911 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:07.755911 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:07.757065 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 29 21:21:07.757065 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 29 21:21:07.757065 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 29 21:21:07.757065 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 29 21:21:07.757065 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 29 21:21:07.757065 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 29 21:21:07.757065 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 21:21:07.757065 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:21:07.761876 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 380/756] 10.222.0.90 () {56 vars in 1111 bytes} [Mon Aug 29 21:21:07 2022] GET /identity/v3/projects/service => generated 88 bytes in 219 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:21:07.765581 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05a80724-773a-4990-96bd-4ca6edb03842 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:07.824102 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be5108e5-3081-4b7a-b0d6-6c4607a4196c None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:07.853900 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:07.885685 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfa03e6d-3445-4043-9f7b-cee8fa516b69 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:07.886615 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfa03e6d-3445-4043-9f7b-cee8fa516b69 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:07.887051 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfa03e6d-3445-4043-9f7b-cee8fa516b69 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:07.887502 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfa03e6d-3445-4043-9f7b-cee8fa516b69 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:07.895823 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfa03e6d-3445-4043-9f7b-cee8fa516b69 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:07.897495 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfa03e6d-3445-4043-9f7b-cee8fa516b69 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:07.911726 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-dfa03e6d-3445-4043-9f7b-cee8fa516b69 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 29 21:21:07.911726 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:07.911726 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:07.911726 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:07.911726 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:07.911726 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:07.911726 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:07.911726 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:07.911726 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:07.911726 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:07.911726 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:07.911726 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:07.911726 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 29 21:21:07.911726 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 29 21:21:07.911726 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 29 21:21:07.911726 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 29 21:21:07.912971 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:07.912971 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:07.912971 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 29 21:21:07.912971 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 21:21:07.912971 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 29 21:21:07.912971 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 21:21:07.912971 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:07.912971 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:07.912971 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:07.912971 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:07.912971 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:07.912971 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:07.912971 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:07.912971 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:07.912971 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:07.912971 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:07.912971 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:07.912971 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:07.914950 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:07.914950 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:07.914950 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 29 21:21:07.914950 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 29 21:21:07.914950 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 29 21:21:07.914950 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 29 21:21:07.914950 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 29 21:21:07.914950 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 29 21:21:07.914950 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 29 21:21:07.914950 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:21:07.915673 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05a80724-773a-4990-96bd-4ca6edb03842 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:07.917694 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 377/757] 10.222.0.90 () {56 vars in 1117 bytes} [Mon Aug 29 21:21:07 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 245 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:21:07.935358 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfa03e6d-3445-4043-9f7b-cee8fa516b69 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:07.958791 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d3f857b-aeff-4d45-a7e9-aa5d82ce61eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:07.959464 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d3f857b-aeff-4d45-a7e9-aa5d82ce61eb None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:07.960007 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d3f857b-aeff-4d45-a7e9-aa5d82ce61eb None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:07.960288 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d3f857b-aeff-4d45-a7e9-aa5d82ce61eb None admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:07.961011 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d3f857b-aeff-4d45-a7e9-aa5d82ce61eb None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:07.961321 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d3f857b-aeff-4d45-a7e9-aa5d82ce61eb None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:07.962798 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d3f857b-aeff-4d45-a7e9-aa5d82ce61eb None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:07.982123 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 381/758] 10.222.0.90 () {56 vars in 1148 bytes} [Mon Aug 29 21:21:07 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:07.984003 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d3f857b-aeff-4d45-a7e9-aa5d82ce61eb None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:08.104937 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d3f857b-aeff-4d45-a7e9-aa5d82ce61eb None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:08.129945 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:08.148227 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbcdb534-c826-4b67-a0d4-decd88fc1bbb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:08.148902 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbcdb534-c826-4b67-a0d4-decd88fc1bbb None admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:08.149188 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbcdb534-c826-4b67-a0d4-decd88fc1bbb None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:08.149469 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbcdb534-c826-4b67-a0d4-decd88fc1bbb None admin] PATH_INFO: `/v3/projects/ec03e112e3d547fbab6faf7b485c3fca/users/a35b5b7dea7b421da056f063d2436ffb/roles/a6e562cdc96c4be691d638990f8403fd` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:08.152118 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbcdb534-c826-4b67-a0d4-decd88fc1bbb None admin] RBAC: Authorizing `identity:create_grant(project_id=ec03e112e3d547fbab6faf7b485c3fca, user_id=a35b5b7dea7b421da056f063d2436ffb, role_id=a6e562cdc96c4be691d638990f8403fd)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:08.153563 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbcdb534-c826-4b67-a0d4-decd88fc1bbb None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:08.175000 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 382/759] 10.222.0.90 () {58 vars in 1439 bytes} [Mon Aug 29 21:21:07 2022] PUT /identity/v3/projects/ec03e112e3d547fbab6faf7b485c3fca/users/a35b5b7dea7b421da056f063d2436ffb/roles/a6e562cdc96c4be691d638990f8403fd => generated 0 bytes in 185 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 21:21:08.210342 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfa03e6d-3445-4043-9f7b-cee8fa516b69 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:08.220177 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:08.270488 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e2898fb-83e4-4d87-859e-74d3d1ef2a65 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:08.271211 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e2898fb-83e4-4d87-859e-74d3d1ef2a65 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:08.271489 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e2898fb-83e4-4d87-859e-74d3d1ef2a65 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:08.271766 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e2898fb-83e4-4d87-859e-74d3d1ef2a65 None admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:08.272516 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e2898fb-83e4-4d87-859e-74d3d1ef2a65 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:08.272824 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e2898fb-83e4-4d87-859e-74d3d1ef2a65 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:08.274341 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e2898fb-83e4-4d87-859e-74d3d1ef2a65 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:08.305302 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 378/760] 10.222.0.90 () {56 vars in 1149 bytes} [Mon Aug 29 21:21:07 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 370 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:08.310767 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbcdb534-c826-4b67-a0d4-decd88fc1bbb None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:08.359478 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e24251b-65a7-4e29-b4dc-4215153dc4bd None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:08.359768 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e24251b-65a7-4e29-b4dc-4215153dc4bd None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:08.359951 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e24251b-65a7-4e29-b4dc-4215153dc4bd None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:08.361723 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 379/761] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:21:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:21:08.367699 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68634509-b342-4f20-a602-2b23e56e2b68 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:08.367982 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68634509-b342-4f20-a602-2b23e56e2b68 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:08.368258 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68634509-b342-4f20-a602-2b23e56e2b68 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:08.388378 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-68634509-b342-4f20-a602-2b23e56e2b68 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:21:08.389578 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-68634509-b342-4f20-a602-2b23e56e2b68 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:21:08.437216 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbcdb534-c826-4b67-a0d4-decd88fc1bbb None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:08.470730 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 380/762] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:21:08 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 21:21:08.483027 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be318b49-4dc3-4375-87ad-bfcd9bb6408a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:08.483656 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be318b49-4dc3-4375-87ad-bfcd9bb6408a None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:08.483940 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be318b49-4dc3-4375-87ad-bfcd9bb6408a None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:08.484290 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be318b49-4dc3-4375-87ad-bfcd9bb6408a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:08.537658 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:08.557970 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be318b49-4dc3-4375-87ad-bfcd9bb6408a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:08.559835 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be318b49-4dc3-4375-87ad-bfcd9bb6408a None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:08.665630 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 383/763] 10.222.0.90 () {58 vars in 1282 bytes} [Mon Aug 29 21:21:08 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 355 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 29 21:21:08.670785 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68634509-b342-4f20-a602-2b23e56e2b68 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:08.802829 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:08.813620 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68634509-b342-4f20-a602-2b23e56e2b68 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:08.862748 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d44e3d7-36cd-46ee-a34e-ca9ff67567e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:08.863399 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d44e3d7-36cd-46ee-a34e-ca9ff67567e4 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:08.863680 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d44e3d7-36cd-46ee-a34e-ca9ff67567e4 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:08.863952 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d44e3d7-36cd-46ee-a34e-ca9ff67567e4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:08.882122 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d44e3d7-36cd-46ee-a34e-ca9ff67567e4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:08.883580 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d44e3d7-36cd-46ee-a34e-ca9ff67567e4 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:08.887499 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be318b49-4dc3-4375-87ad-bfcd9bb6408a None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:08.907167 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-8d44e3d7-36cd-46ee-a34e-ca9ff67567e4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:21:08.907167 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:08.907167 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:08.907167 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:08.907167 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:08.907167 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:08.907167 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:08.907167 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:08.907167 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:08.907167 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:08.907167 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:08.907167 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:08.907167 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 21:21:08.907167 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 21:21:08.907167 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 21:21:08.907167 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 21:21:08.908254 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:08.908254 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:08.908254 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:08.908254 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:08.908254 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:08.908254 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:08.908254 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:08.908254 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:08.908254 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:08.908254 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:08.908254 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:08.908254 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:08.908254 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:08.908254 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:08.908254 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:08.908254 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:08.908254 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 21:21:08.909722 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 21:21:08.909722 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:21:08.909722 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:21:08.912029 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 381/764] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:21:08 2022] GET /identity/v3/domains/Default => generated 87 bytes in 242 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:21:08.918979 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d44e3d7-36cd-46ee-a34e-ca9ff67567e4 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:08.982222 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:09.004331 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be318b49-4dc3-4375-87ad-bfcd9bb6408a None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:09.045020 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48d6eb25-505f-48d2-b46d-4cc405f04d05 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:09.045587 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48d6eb25-505f-48d2-b46d-4cc405f04d05 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:09.049807 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48d6eb25-505f-48d2-b46d-4cc405f04d05 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:09.049807 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48d6eb25-505f-48d2-b46d-4cc405f04d05 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:09.049807 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48d6eb25-505f-48d2-b46d-4cc405f04d05 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:09.049807 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48d6eb25-505f-48d2-b46d-4cc405f04d05 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:09.057114 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-48d6eb25-505f-48d2-b46d-4cc405f04d05 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 29 21:21:09.057114 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:09.057114 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 29 21:21:09.057114 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 29 21:21:09.057114 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 29 21:21:09.057114 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 29 21:21:09.057114 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 29 21:21:09.057114 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 21:21:09.057114 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 29 21:21:09.057114 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:21:09.057114 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 29 21:21:09.057114 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:21:09.057114 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:09.057114 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:09.057114 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:09.057114 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:09.057114 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:09.058182 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:09.058182 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:09.058182 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:09.058182 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:09.058182 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:09.058182 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:09.058182 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 29 21:21:09.058182 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 29 21:21:09.058182 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 29 21:21:09.058182 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 29 21:21:09.058182 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:09.058182 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:09.058182 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:09.058182 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:09.058182 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:09.058182 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:09.059048 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:09.059048 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:09.059048 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:09.059048 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:09.059048 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:09.059048 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:09.059048 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:09.059048 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:09.059048 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:09.059048 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:09.059048 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 29 21:21:09.059048 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 21:21:09.059048 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 29 21:21:09.059048 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:21:09.059048 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 384/765] 10.222.0.90 () {56 vars in 1129 bytes} [Mon Aug 29 21:21:08 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 29 21:21:09.077064 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48d6eb25-505f-48d2-b46d-4cc405f04d05 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:09.108065 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d44e3d7-36cd-46ee-a34e-ca9ff67567e4 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:09.173300 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e43dc611-2d8b-44b3-aff4-05edcf349ecd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:09.174545 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e43dc611-2d8b-44b3-aff4-05edcf349ecd None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:09.175043 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e43dc611-2d8b-44b3-aff4-05edcf349ecd None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:09.175525 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e43dc611-2d8b-44b3-aff4-05edcf349ecd None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:09.176399 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e43dc611-2d8b-44b3-aff4-05edcf349ecd None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:09.176913 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e43dc611-2d8b-44b3-aff4-05edcf349ecd None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:09.178537 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e43dc611-2d8b-44b3-aff4-05edcf349ecd None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:09.197457 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 382/766] 10.222.0.90 () {56 vars in 1109 bytes} [Mon Aug 29 21:21:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 279 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:09.204729 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e43dc611-2d8b-44b3-aff4-05edcf349ecd None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:09.205066 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48d6eb25-505f-48d2-b46d-4cc405f04d05 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:09.213648 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:09.244290 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46620cc0-4cc9-4bd6-8e48-c200c520bd42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:09.244910 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46620cc0-4cc9-4bd6-8e48-c200c520bd42 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:09.245192 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46620cc0-4cc9-4bd6-8e48-c200c520bd42 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:09.245469 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46620cc0-4cc9-4bd6-8e48-c200c520bd42 None admin] PATH_INFO: `/v3/services` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:09.246156 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46620cc0-4cc9-4bd6-8e48-c200c520bd42 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:09.246463 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46620cc0-4cc9-4bd6-8e48-c200c520bd42 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:09.248062 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46620cc0-4cc9-4bd6-8e48-c200c520bd42 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:09.253447 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 385/767] 10.222.0.90 () {56 vars in 1124 bytes} [Mon Aug 29 21:21:09 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:09.259411 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46620cc0-4cc9-4bd6-8e48-c200c520bd42 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:09.281811 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:09.387195 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46620cc0-4cc9-4bd6-8e48-c200c520bd42 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:09.420111 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e43dc611-2d8b-44b3-aff4-05edcf349ecd None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:09.425398 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70dee78a-1bb8-41d4-84e0-06d345fbf1af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:09.433816 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70dee78a-1bb8-41d4-84e0-06d345fbf1af None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:09.433816 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70dee78a-1bb8-41d4-84e0-06d345fbf1af None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:09.433816 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70dee78a-1bb8-41d4-84e0-06d345fbf1af None admin] PATH_INFO: `/v3/services` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:09.433816 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70dee78a-1bb8-41d4-84e0-06d345fbf1af None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:09.433816 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70dee78a-1bb8-41d4-84e0-06d345fbf1af None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:09.433816 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70dee78a-1bb8-41d4-84e0-06d345fbf1af None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:09.435214 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 386/768] 10.222.0.90 () {56 vars in 1124 bytes} [Mon Aug 29 21:21:09 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:09.452385 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70dee78a-1bb8-41d4-84e0-06d345fbf1af None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:09.473032 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7bdc1b7-662b-44fa-893e-d5f5026300c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:09.474025 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7bdc1b7-662b-44fa-893e-d5f5026300c6 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:09.474493 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7bdc1b7-662b-44fa-893e-d5f5026300c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:09.474948 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7bdc1b7-662b-44fa-893e-d5f5026300c6 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:09.482267 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7bdc1b7-662b-44fa-893e-d5f5026300c6 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:09.484015 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7bdc1b7-662b-44fa-893e-d5f5026300c6 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:09.496118 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-a7bdc1b7-662b-44fa-893e-d5f5026300c6 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 21:21:09.496118 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:09.496118 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:09.496118 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:09.496118 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:09.496118 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:09.496118 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:09.496118 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:09.496118 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:09.496118 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:09.496118 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:09.496118 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:09.496118 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 29 21:21:09.496118 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 29 21:21:09.496118 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 29 21:21:09.497082 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 29 21:21:09.497082 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:09.497082 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:09.497082 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:09.497082 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:09.497082 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:09.497082 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:09.497082 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:09.497082 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:09.497082 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:09.497082 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:09.497082 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:09.497082 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:09.497082 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:09.497082 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:09.497082 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:09.497082 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:09.498064 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 29 21:21:09.498064 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 29 21:21:09.498064 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 29 21:21:09.498064 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 29 21:21:09.498064 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 29 21:21:09.498064 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 29 21:21:09.498064 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 21:21:09.498064 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:21:09.500444 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 383/769] 10.222.0.90 () {56 vars in 1111 bytes} [Mon Aug 29 21:21:09 2022] GET /identity/v3/projects/service => generated 88 bytes in 296 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:21:09.510795 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7bdc1b7-662b-44fa-893e-d5f5026300c6 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:09.580972 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70dee78a-1bb8-41d4-84e0-06d345fbf1af None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:09.594164 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:09.599765 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:09.646935 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7bdc1b7-662b-44fa-893e-d5f5026300c6 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:09.681285 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d3092f1-af0b-4d7d-84c3-bce5e1f3ad62 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:09.681862 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d3092f1-af0b-4d7d-84c3-bce5e1f3ad62 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:09.682238 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d3092f1-af0b-4d7d-84c3-bce5e1f3ad62 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:09.682522 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d3092f1-af0b-4d7d-84c3-bce5e1f3ad62 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:09.683355 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d3092f1-af0b-4d7d-84c3-bce5e1f3ad62 None admin] RBAC: Adding query filter params (interface=public, service_id=c5e4b519fa2c4e8990807f473c44a2fc, region_id=RegionOne) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:09.683674 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d3092f1-af0b-4d7d-84c3-bce5e1f3ad62 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:09.684507 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac68a762-6b69-40a4-b4f8-73f644f78b3e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:09.685031 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d3092f1-af0b-4d7d-84c3-bce5e1f3ad62 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:09.685571 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac68a762-6b69-40a4-b4f8-73f644f78b3e None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:09.686096 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac68a762-6b69-40a4-b4f8-73f644f78b3e None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:09.686572 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac68a762-6b69-40a4-b4f8-73f644f78b3e None admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:09.687463 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac68a762-6b69-40a4-b4f8-73f644f78b3e None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:09.687969 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac68a762-6b69-40a4-b4f8-73f644f78b3e None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:09.689799 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac68a762-6b69-40a4-b4f8-73f644f78b3e None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:09.703106 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 384/770] 10.222.0.90 () {56 vars in 1148 bytes} [Mon Aug 29 21:21:09 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:09.704034 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 387/771] 10.222.0.90 () {56 vars in 1251 bytes} [Mon Aug 29 21:21:09 2022] GET /identity/v3/endpoints?service_id=c5e4b519fa2c4e8990807f473c44a2fc&interface=public®ion_id=RegionOne => generated 194 bytes in 252 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:09.709215 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac68a762-6b69-40a4-b4f8-73f644f78b3e None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:09.711824 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d3092f1-af0b-4d7d-84c3-bce5e1f3ad62 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:09.816202 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac68a762-6b69-40a4-b4f8-73f644f78b3e None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:09.850177 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:09.864747 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-001b092d-2ef3-44a2-8512-7abe3dffdbdb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:09.866060 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-001b092d-2ef3-44a2-8512-7abe3dffdbdb None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:09.866556 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-001b092d-2ef3-44a2-8512-7abe3dffdbdb None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:09.867114 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-001b092d-2ef3-44a2-8512-7abe3dffdbdb None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:09.868049 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-001b092d-2ef3-44a2-8512-7abe3dffdbdb None admin] RBAC: Adding query filter params (role.id=a94bdb4c5796428baf63644c062dfcb2, scope.project.id=ec03e112e3d547fbab6faf7b485c3fca, user.id=0c0b0da87b124de1be00faec4bbdb404) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:09.868605 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-001b092d-2ef3-44a2-8512-7abe3dffdbdb None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:09.870279 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-001b092d-2ef3-44a2-8512-7abe3dffdbdb None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:09.876720 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 385/772] 10.222.0.90 () {56 vars in 1374 bytes} [Mon Aug 29 21:21:09 2022] GET /identity/v3/role_assignments?user.id=0c0b0da87b124de1be00faec4bbdb404&scope.project.id=ec03e112e3d547fbab6faf7b485c3fca&role.id=a94bdb4c5796428baf63644c062dfcb2 => generated 259 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:09.913273 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d3092f1-af0b-4d7d-84c3-bce5e1f3ad62 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:09.961864 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-770ed163-b86f-4e2b-aaf4-9a15bae8bcc3 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:09.962162 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-770ed163-b86f-4e2b-aaf4-9a15bae8bcc3 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:09.962350 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-770ed163-b86f-4e2b-aaf4-9a15bae8bcc3 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:09.963783 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 386/773] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:21:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 29 21:21:09.964871 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a5d4635-2219-4286-8147-635b26ca4f4e None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:09.965156 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a5d4635-2219-4286-8147-635b26ca4f4e None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:09.965434 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a5d4635-2219-4286-8147-635b26ca4f4e None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:09.967016 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 387/774] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:21:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:21:09.975993 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-137c0215-2ee5-4d86-9fde-f28d727526c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:09.976239 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e100d8-3daf-430a-9730-eb3439639597 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:09.976463 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e100d8-3daf-430a-9730-eb3439639597 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:09.976631 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-137c0215-2ee5-4d86-9fde-f28d727526c4 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:09.976726 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e100d8-3daf-430a-9730-eb3439639597 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:09.976906 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-137c0215-2ee5-4d86-9fde-f28d727526c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:09.977185 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-137c0215-2ee5-4d86-9fde-f28d727526c4 None admin] PATH_INFO: `/v3/services` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:09.977843 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-137c0215-2ee5-4d86-9fde-f28d727526c4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:09.979261 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-137c0215-2ee5-4d86-9fde-f28d727526c4 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:09.981463 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:09.985981 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 388/775] 10.222.0.90 () {56 vars in 1087 bytes} [Mon Aug 29 21:21:09 2022] GET /identity/v3/services => generated 772 bytes in 274 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:21:09.988033 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36fa624b-8775-4f63-8677-ac7f06426a3c None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:09.988291 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36fa624b-8775-4f63-8677-ac7f06426a3c None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:09.988740 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36fa624b-8775-4f63-8677-ac7f06426a3c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:10.007202 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-e0e100d8-3daf-430a-9730-eb3439639597 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:21:10.008724 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-e0e100d8-3daf-430a-9730-eb3439639597 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:21:10.009547 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-36fa624b-8775-4f63-8677-ac7f06426a3c None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:21:10.010905 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-36fa624b-8775-4f63-8677-ac7f06426a3c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:21:10.093216 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 389/776] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:21:09 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 21:21:10.156020 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 388/777] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:21:09 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 29 21:21:10.398278 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36fa624b-8775-4f63-8677-ac7f06426a3c None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:10.520955 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0e100d8-3daf-430a-9730-eb3439639597 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:10.536210 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36fa624b-8775-4f63-8677-ac7f06426a3c None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:10.574271 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:10.578470 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9106de93-89c4-47e5-a86f-2e2734d17389 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:10.579652 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9106de93-89c4-47e5-a86f-2e2734d17389 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:10.580817 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9106de93-89c4-47e5-a86f-2e2734d17389 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:10.582153 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9106de93-89c4-47e5-a86f-2e2734d17389 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:10.609569 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9106de93-89c4-47e5-a86f-2e2734d17389 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 21:21:10.621430 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-9106de93-89c4-47e5-a86f-2e2734d17389 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 21:21:10.621430 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:10.621430 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:10.621430 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:10.621430 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:10.621430 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:10.621430 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:10.621430 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:10.621430 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:10.621430 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:10.621430 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:10.621430 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:10.621430 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 21:21:10.621430 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 21:21:10.621430 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 21:21:10.621430 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 21:21:10.622500 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 21:21:10.622500 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 21:21:10.622500 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 21:21:10.622500 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 21:21:10.622500 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:10.622500 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:10.622500 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:10.622500 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:10.622500 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:10.622500 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:10.622500 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:10.622500 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:10.622500 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:10.622500 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:10.622500 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:10.622500 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:10.623387 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:10.623387 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:10.623387 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:10.623387 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:10.623387 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 21:21:10.623387 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 21:21:10.623387 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 21:21:10.623387 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 21:21:10.623387 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 21:21:10.623387 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 21:21:10.623387 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 21:21:10.623387 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:21:10.628569 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 390/778] 10.222.0.90 () {56 vars in 1096 bytes} [Mon Aug 29 21:21:10 2022] GET /identity/v3/roles/admin => generated 83 bytes in 231 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:21:10.634626 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9106de93-89c4-47e5-a86f-2e2734d17389 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:10.715540 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0e100d8-3daf-430a-9730-eb3439639597 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:10.736259 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:10.772248 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c354cdbc-0d34-4175-8b59-c89266de3e32 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:10.772895 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c354cdbc-0d34-4175-8b59-c89266de3e32 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:10.773182 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c354cdbc-0d34-4175-8b59-c89266de3e32 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:10.773516 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c354cdbc-0d34-4175-8b59-c89266de3e32 None admin] PATH_INFO: `/v3/services/network` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:10.774137 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c354cdbc-0d34-4175-8b59-c89266de3e32 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:10.779757 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c354cdbc-0d34-4175-8b59-c89266de3e32 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:10.798321 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-c354cdbc-0d34-4175-8b59-c89266de3e32 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Aug 29 21:21:10.798321 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:10.798321 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 29 21:21:10.798321 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 29 21:21:10.798321 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 29 21:21:10.798321 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 29 21:21:10.798321 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 29 21:21:10.798321 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 21:21:10.798321 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Aug 29 21:21:10.798321 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:21:10.798321 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 29 21:21:10.798321 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:21:10.798321 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:10.798321 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:10.798321 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:10.798321 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:10.798321 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:10.800178 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:10.800178 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:10.800178 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:10.800178 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:10.800178 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:10.800178 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:10.800178 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 29 21:21:10.800178 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 29 21:21:10.800178 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 29 21:21:10.800178 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 29 21:21:10.800178 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:10.800178 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:10.800178 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:10.800178 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:10.800178 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:10.800178 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:10.801036 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:10.801036 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:10.801036 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:10.801036 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:10.801036 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:10.801036 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:10.801036 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:10.801036 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:10.801036 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:10.801036 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:10.801036 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 29 21:21:10.801036 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 21:21:10.801036 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Aug 29 21:21:10.801036 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:21:10.813174 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 389/779] 10.222.0.90 () {56 vars in 1111 bytes} [Mon Aug 29 21:21:10 2022] GET /identity/v3/services/network => generated 88 bytes in 292 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:21:10.815378 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c354cdbc-0d34-4175-8b59-c89266de3e32 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:10.882985 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9106de93-89c4-47e5-a86f-2e2734d17389 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:10.959427 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afd2a1dd-b5eb-4c52-a833-71b1b3fe568f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:10.960055 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd2a1dd-b5eb-4c52-a833-71b1b3fe568f None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:10.960337 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd2a1dd-b5eb-4c52-a833-71b1b3fe568f None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:10.960614 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd2a1dd-b5eb-4c52-a833-71b1b3fe568f None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:10.961295 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afd2a1dd-b5eb-4c52-a833-71b1b3fe568f None admin] RBAC: Adding query filter params (name=admin) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:10.961605 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afd2a1dd-b5eb-4c52-a833-71b1b3fe568f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:10.966142 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afd2a1dd-b5eb-4c52-a833-71b1b3fe568f None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:10.986189 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 391/780] 10.222.0.90 () {56 vars in 1099 bytes} [Mon Aug 29 21:21:10 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 352 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:21:10.990270 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37246c08-0eb8-4de6-bedc-504b29e6ecb2 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:10.990559 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37246c08-0eb8-4de6-bedc-504b29e6ecb2 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:10.990849 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37246c08-0eb8-4de6-bedc-504b29e6ecb2 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:10.992388 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 392/781] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:21:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:21:10.993188 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37246c08-0eb8-4de6-bedc-504b29e6ecb2 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:11.012132 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c354cdbc-0d34-4175-8b59-c89266de3e32 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:11.051340 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdf1f30e-d386-43f8-a8a0-2e24117a7646 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:11.051969 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf1f30e-d386-43f8-a8a0-2e24117a7646 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:11.052252 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf1f30e-d386-43f8-a8a0-2e24117a7646 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:11.052638 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf1f30e-d386-43f8-a8a0-2e24117a7646 None admin] PATH_INFO: `/v3/services` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:11.053280 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdf1f30e-d386-43f8-a8a0-2e24117a7646 None admin] RBAC: Adding query filter params (name=network) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:11.053585 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdf1f30e-d386-43f8-a8a0-2e24117a7646 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:11.055104 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdf1f30e-d386-43f8-a8a0-2e24117a7646 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:11.060529 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 390/782] 10.222.0.90 () {56 vars in 1112 bytes} [Mon Aug 29 21:21:10 2022] GET /identity/v3/services?name=network => generated 124 bytes in 246 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:11.061694 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c6002fa-51ea-4600-9d34-7f87e05ce42e None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:11.062020 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c6002fa-51ea-4600-9d34-7f87e05ce42e None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:11.062298 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c6002fa-51ea-4600-9d34-7f87e05ce42e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:11.067786 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:11.084146 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-3c6002fa-51ea-4600-9d34-7f87e05ce42e None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:21:11.085768 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-3c6002fa-51ea-4600-9d34-7f87e05ce42e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:21:11.132438 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37246c08-0eb8-4de6-bedc-504b29e6ecb2 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:11.144756 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:11.177280 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 391/783] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:21:11 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 21:21:11.178158 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b4fe479-adf8-43a3-9a5b-8707e5edba4b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:11.178630 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c6002fa-51ea-4600-9d34-7f87e05ce42e None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:11.178923 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b4fe479-adf8-43a3-9a5b-8707e5edba4b None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:11.179218 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b4fe479-adf8-43a3-9a5b-8707e5edba4b None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:11.179522 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b4fe479-adf8-43a3-9a5b-8707e5edba4b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:11.194245 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b4fe479-adf8-43a3-9a5b-8707e5edba4b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:11.195906 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b4fe479-adf8-43a3-9a5b-8707e5edba4b None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:11.215593 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-3b4fe479-adf8-43a3-9a5b-8707e5edba4b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:21:11.215593 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:11.215593 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:11.215593 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:11.215593 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:11.215593 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:11.215593 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:11.215593 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:11.215593 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:11.215593 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:11.215593 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:11.215593 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:11.215593 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 21:21:11.215593 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 21:21:11.215593 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 21:21:11.215593 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 21:21:11.216849 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:11.216849 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:11.216849 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:11.216849 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:11.216849 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:11.216849 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:11.216849 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:11.216849 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:11.216849 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:11.216849 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:11.216849 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:11.216849 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:11.216849 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:11.216849 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:11.216849 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:11.216849 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:11.216849 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 21:21:11.218034 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 21:21:11.218034 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:21:11.218034 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:21:11.218034 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 393/784] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:21:10 2022] GET /identity/v3/domains/Default => generated 87 bytes in 224 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:21:11.222943 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b4fe479-adf8-43a3-9a5b-8707e5edba4b None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:11.297924 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c6002fa-51ea-4600-9d34-7f87e05ce42e None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:11.347484 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cefbd2b7-a226-4658-a247-74be3cad4dff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:11.348528 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cefbd2b7-a226-4658-a247-74be3cad4dff None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:11.349016 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cefbd2b7-a226-4658-a247-74be3cad4dff None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:11.349493 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cefbd2b7-a226-4658-a247-74be3cad4dff None admin] PATH_INFO: `/v3/services` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:11.350470 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cefbd2b7-a226-4658-a247-74be3cad4dff None admin] RBAC: Adding query filter params (type=network) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:11.350994 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cefbd2b7-a226-4658-a247-74be3cad4dff None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:11.352636 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cefbd2b7-a226-4658-a247-74be3cad4dff None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:11.358383 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 392/785] 10.222.0.90 () {56 vars in 1112 bytes} [Mon Aug 29 21:21:11 2022] GET /identity/v3/services?type=network => generated 350 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:11.367986 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cefbd2b7-a226-4658-a247-74be3cad4dff None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:11.373950 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b4fe479-adf8-43a3-9a5b-8707e5edba4b None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:11.376333 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:11.418345 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:11.446411 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f66a64f2-a55a-4f0e-a8ff-26ba24fab821 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:11.447040 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f66a64f2-a55a-4f0e-a8ff-26ba24fab821 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:11.447337 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f66a64f2-a55a-4f0e-a8ff-26ba24fab821 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:11.447598 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f66a64f2-a55a-4f0e-a8ff-26ba24fab821 None admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:11.448253 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f66a64f2-a55a-4f0e-a8ff-26ba24fab821 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:11.448559 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f66a64f2-a55a-4f0e-a8ff-26ba24fab821 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:11.450011 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f66a64f2-a55a-4f0e-a8ff-26ba24fab821 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:11.472108 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 394/786] 10.222.0.90 () {56 vars in 1109 bytes} [Mon Aug 29 21:21:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 249 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:11.478307 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f66a64f2-a55a-4f0e-a8ff-26ba24fab821 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:11.487179 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cefbd2b7-a226-4658-a247-74be3cad4dff None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:11.539763 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7338c45-fc42-4da3-aee9-e53bb22b47d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:11.540108 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7338c45-fc42-4da3-aee9-e53bb22b47d8 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:11.540349 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7338c45-fc42-4da3-aee9-e53bb22b47d8 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:11.540616 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7338c45-fc42-4da3-aee9-e53bb22b47d8 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:11.541316 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7338c45-fc42-4da3-aee9-e53bb22b47d8 None admin] RBAC: Adding query filter params (interface=public, service_id=8e2cbe38bc924409b570733a9ad6fc4c, region_id=RegionOne) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:11.541621 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7338c45-fc42-4da3-aee9-e53bb22b47d8 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:11.543018 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7338c45-fc42-4da3-aee9-e53bb22b47d8 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:11.561049 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 393/787] 10.222.0.90 () {56 vars in 1251 bytes} [Mon Aug 29 21:21:11 2022] GET /identity/v3/endpoints?service_id=8e2cbe38bc924409b570733a9ad6fc4c&interface=public®ion_id=RegionOne => generated 194 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:11.566760 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7338c45-fc42-4da3-aee9-e53bb22b47d8 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:11.619689 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f66a64f2-a55a-4f0e-a8ff-26ba24fab821 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:11.665575 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-618277b9-f584-4b82-9ab1-7eaef44f4dd6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:11.667153 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-618277b9-f584-4b82-9ab1-7eaef44f4dd6 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:11.667625 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-618277b9-f584-4b82-9ab1-7eaef44f4dd6 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:11.668081 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-618277b9-f584-4b82-9ab1-7eaef44f4dd6 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:11.679696 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-618277b9-f584-4b82-9ab1-7eaef44f4dd6 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:11.682232 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-618277b9-f584-4b82-9ab1-7eaef44f4dd6 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:11.700358 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-618277b9-f584-4b82-9ab1-7eaef44f4dd6 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Aug 29 21:21:11.700358 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:11.700358 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:11.700358 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:11.700358 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:11.700358 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:11.700358 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:11.700358 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:11.700358 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:11.700358 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:11.700358 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:11.700358 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:11.700358 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 29 21:21:11.700358 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 29 21:21:11.700358 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 29 21:21:11.700358 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 29 21:21:11.701368 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:11.701368 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:11.701368 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 29 21:21:11.701368 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 21:21:11.701368 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 29 21:21:11.701368 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 21:21:11.701368 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:11.701368 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:11.701368 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:11.701368 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:11.701368 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:11.701368 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:11.701368 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:11.701368 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:11.701368 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:11.701368 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:11.701368 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:11.701368 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:11.702420 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:11.702420 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:11.702420 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 29 21:21:11.702420 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 29 21:21:11.702420 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 29 21:21:11.702420 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 29 21:21:11.702420 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 29 21:21:11.702420 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 29 21:21:11.702420 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Aug 29 21:21:11.702420 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:21:11.704192 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 395/788] 10.222.0.90 () {56 vars in 1096 bytes} [Mon Aug 29 21:21:11 2022] GET /identity/v3/users/swift => generated 83 bytes in 225 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 29 21:21:11.704954 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-618277b9-f584-4b82-9ab1-7eaef44f4dd6 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:11.732899 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:11.738085 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7338c45-fc42-4da3-aee9-e53bb22b47d8 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:11.741923 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:11.785495 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42b95337-f93e-4b15-ae47-ce6379b2419a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:11.786511 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42b95337-f93e-4b15-ae47-ce6379b2419a None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:11.787027 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42b95337-f93e-4b15-ae47-ce6379b2419a None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:11.787522 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42b95337-f93e-4b15-ae47-ce6379b2419a None admin] PATH_INFO: `/v3/services` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:11.788340 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42b95337-f93e-4b15-ae47-ce6379b2419a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:11.790477 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42b95337-f93e-4b15-ae47-ce6379b2419a None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:11.798696 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 394/789] 10.222.0.90 () {56 vars in 1087 bytes} [Mon Aug 29 21:21:11 2022] GET /identity/v3/services => generated 772 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:21:11.800032 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42b95337-f93e-4b15-ae47-ce6379b2419a None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:11.842295 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-618277b9-f584-4b82-9ab1-7eaef44f4dd6 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:11.929504 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42b95337-f93e-4b15-ae47-ce6379b2419a None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:11.934713 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a4ea965-f769-4701-b118-699ec2766f8f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:11.935378 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a4ea965-f769-4701-b118-699ec2766f8f None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:11.935661 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a4ea965-f769-4701-b118-699ec2766f8f None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:11.935939 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a4ea965-f769-4701-b118-699ec2766f8f None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:11.966106 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a4ea965-f769-4701-b118-699ec2766f8f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 21:21:11.972055 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-880ed0b2-1b32-44f0-be67-1b2568c2bead None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:11.973028 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-880ed0b2-1b32-44f0-be67-1b2568c2bead None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:11.973522 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-880ed0b2-1b32-44f0-be67-1b2568c2bead None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:11.974071 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-880ed0b2-1b32-44f0-be67-1b2568c2bead None admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:11.975027 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-880ed0b2-1b32-44f0-be67-1b2568c2bead None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:11.975563 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-880ed0b2-1b32-44f0-be67-1b2568c2bead None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:11.975832 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-2a4ea965-f769-4701-b118-699ec2766f8f None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 21:21:11.975832 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:11.975832 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:11.975832 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:11.975832 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:11.975832 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:11.975832 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:11.975832 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:11.975832 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:11.975832 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:11.975832 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:11.975832 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:11.975832 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 21:21:11.975832 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 21:21:11.975832 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 21:21:11.975832 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 21:21:11.976754 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 21:21:11.976754 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 21:21:11.976754 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 21:21:11.976754 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 21:21:11.976754 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:11.976754 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:11.976754 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:11.976754 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:11.976754 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:11.976754 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:11.976754 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:11.976754 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:11.976754 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:11.976754 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:11.976754 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:11.976754 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:11.977624 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:11.977624 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:11.977624 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:11.977624 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:11.977624 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 21:21:11.977624 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 21:21:11.977624 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 21:21:11.977624 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 21:21:11.977624 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 21:21:11.977624 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 21:21:11.977624 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 21:21:11.977624 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:21:11.977624 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 396/790] 10.222.0.90 () {56 vars in 1096 bytes} [Mon Aug 29 21:21:11 2022] GET /identity/v3/roles/admin => generated 83 bytes in 272 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:21:11.980424 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-880ed0b2-1b32-44f0-be67-1b2568c2bead None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:11.983283 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a4ea965-f769-4701-b118-699ec2766f8f None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:12.003972 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 395/791] 10.222.0.90 () {56 vars in 1135 bytes} [Mon Aug 29 21:21:11 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:12.009113 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-880ed0b2-1b32-44f0-be67-1b2568c2bead None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:12.048476 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:12.115366 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:12.135193 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a4ea965-f769-4701-b118-699ec2766f8f None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:12.137302 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-880ed0b2-1b32-44f0-be67-1b2568c2bead None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:12.177596 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e26f1e4-dc25-485d-99ec-b5ae3c2f7ae4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:12.178502 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e26f1e4-dc25-485d-99ec-b5ae3c2f7ae4 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:12.178834 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e26f1e4-dc25-485d-99ec-b5ae3c2f7ae4 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:12.179200 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e26f1e4-dc25-485d-99ec-b5ae3c2f7ae4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:12.183450 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab62e867-04f2-48ca-a8ac-1637a40e772b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:12.184268 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab62e867-04f2-48ca-a8ac-1637a40e772b None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:12.184699 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab62e867-04f2-48ca-a8ac-1637a40e772b None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:12.185122 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab62e867-04f2-48ca-a8ac-1637a40e772b None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:12.185977 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab62e867-04f2-48ca-a8ac-1637a40e772b None admin] RBAC: Adding query filter params (name=admin) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:12.186448 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab62e867-04f2-48ca-a8ac-1637a40e772b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:12.188544 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab62e867-04f2-48ca-a8ac-1637a40e772b None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:12.192036 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e26f1e4-dc25-485d-99ec-b5ae3c2f7ae4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:12.193772 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e26f1e4-dc25-485d-99ec-b5ae3c2f7ae4 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:12.205717 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 397/792] 10.222.0.90 () {56 vars in 1099 bytes} [Mon Aug 29 21:21:11 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:21:12.205717 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5878c321-5594-4041-ad53-39223bdf7265 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:12.205717 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5878c321-5594-4041-ad53-39223bdf7265 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:12.205717 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5878c321-5594-4041-ad53-39223bdf7265 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:12.209854 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 398/793] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:21:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:21:12.211487 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5878c321-5594-4041-ad53-39223bdf7265 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:12.217924 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-2e26f1e4-dc25-485d-99ec-b5ae3c2f7ae4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:21:12.217924 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:12.217924 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:12.217924 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:12.217924 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:12.217924 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:12.217924 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:12.217924 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:12.217924 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:12.217924 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:12.217924 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:12.217924 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:12.217924 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 21:21:12.217924 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 21:21:12.217924 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 21:21:12.217924 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 21:21:12.218941 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:12.218941 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:12.218941 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:12.218941 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:12.218941 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:12.218941 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:12.218941 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:12.218941 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:12.218941 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:12.218941 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:12.218941 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:12.218941 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:12.218941 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:12.218941 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:12.218941 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:12.218941 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:12.218941 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 21:21:12.220440 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 21:21:12.220440 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:21:12.220440 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:21:12.220440 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 396/794] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:21:12 2022] GET /identity/v3/domains/Default => generated 87 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 29 21:21:12.220741 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ddaabbf-d435-438e-8f60-0d987ff56248 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:12.221302 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ddaabbf-d435-438e-8f60-0d987ff56248 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:12.221462 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ddaabbf-d435-438e-8f60-0d987ff56248 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:12.242361 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-4ddaabbf-d435-438e-8f60-0d987ff56248 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:21:12.243596 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-4ddaabbf-d435-438e-8f60-0d987ff56248 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:21:12.322145 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 397/795] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:21:12 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 21:21:12.323084 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ddaabbf-d435-438e-8f60-0d987ff56248 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:12.337551 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:12.358809 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5878c321-5594-4041-ad53-39223bdf7265 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:12.406899 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb895538-a185-4a00-8343-dc646a9e941a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:12.407895 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb895538-a185-4a00-8343-dc646a9e941a None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:12.408885 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb895538-a185-4a00-8343-dc646a9e941a None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:12.409939 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb895538-a185-4a00-8343-dc646a9e941a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:12.425850 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb895538-a185-4a00-8343-dc646a9e941a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:12.428282 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb895538-a185-4a00-8343-dc646a9e941a None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:12.438636 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ddaabbf-d435-438e-8f60-0d987ff56248 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:12.444581 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:12.457377 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-bb895538-a185-4a00-8343-dc646a9e941a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:21:12.457377 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:12.457377 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:12.457377 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:12.457377 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:12.457377 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:12.457377 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:12.457377 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:12.457377 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:12.457377 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:12.457377 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:12.457377 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:12.457377 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 21:21:12.457377 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 21:21:12.457377 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 21:21:12.457377 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 21:21:12.458490 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:12.458490 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:12.458490 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:12.458490 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:12.458490 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:12.458490 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:12.458490 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:12.458490 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:12.458490 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:12.458490 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:12.458490 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:12.458490 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:12.458490 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:12.458490 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:12.458490 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:12.458490 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:12.458490 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 21:21:12.459411 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 21:21:12.459411 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:21:12.459411 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:21:12.461501 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 399/796] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:21:12 2022] GET /identity/v3/domains/Default => generated 87 bytes in 251 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:21:12.462937 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e943ffb-56b7-4479-b9b2-70c397fe08c1 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:12.463480 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e943ffb-56b7-4479-b9b2-70c397fe08c1 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:12.463957 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e943ffb-56b7-4479-b9b2-70c397fe08c1 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:12.465685 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 400/797] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:21:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:21:12.466938 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e943ffb-56b7-4479-b9b2-70c397fe08c1 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:12.478812 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fcc05eb-f47e-45ef-9a36-e761e9edfc3b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:12.479432 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fcc05eb-f47e-45ef-9a36-e761e9edfc3b None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:12.479709 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fcc05eb-f47e-45ef-9a36-e761e9edfc3b None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:12.479982 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fcc05eb-f47e-45ef-9a36-e761e9edfc3b None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:12.480624 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fcc05eb-f47e-45ef-9a36-e761e9edfc3b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:12.480923 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fcc05eb-f47e-45ef-9a36-e761e9edfc3b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:12.482313 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fcc05eb-f47e-45ef-9a36-e761e9edfc3b None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:12.494848 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 398/798] 10.222.0.90 () {56 vars in 1109 bytes} [Mon Aug 29 21:21:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:12.496007 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc012fc7-1ac3-4765-a5dd-b261829e5677 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:12.496288 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc012fc7-1ac3-4765-a5dd-b261829e5677 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:12.496559 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc012fc7-1ac3-4765-a5dd-b261829e5677 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:12.521960 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-fc012fc7-1ac3-4765-a5dd-b261829e5677 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:21:12.524205 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-fc012fc7-1ac3-4765-a5dd-b261829e5677 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:21:12.595434 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e943ffb-56b7-4479-b9b2-70c397fe08c1 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:12.607608 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:12.617928 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 399/799] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:21:12 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 21:21:12.618779 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc012fc7-1ac3-4765-a5dd-b261829e5677 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:12.647351 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3be8144-0f44-472e-b8d6-ad035fe7bee3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:12.648996 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3be8144-0f44-472e-b8d6-ad035fe7bee3 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:12.649630 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3be8144-0f44-472e-b8d6-ad035fe7bee3 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:12.650216 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3be8144-0f44-472e-b8d6-ad035fe7bee3 None admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:12.651317 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3be8144-0f44-472e-b8d6-ad035fe7bee3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:12.652204 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3be8144-0f44-472e-b8d6-ad035fe7bee3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:12.654101 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3be8144-0f44-472e-b8d6-ad035fe7bee3 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:12.671466 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 401/800] 10.222.0.90 () {56 vars in 1109 bytes} [Mon Aug 29 21:21:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:12.673179 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3be8144-0f44-472e-b8d6-ad035fe7bee3 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:12.740161 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:12.746290 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc012fc7-1ac3-4765-a5dd-b261829e5677 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:12.784947 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-952ec722-837f-485f-a63f-d6c6b2d50af5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:12.785678 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-952ec722-837f-485f-a63f-d6c6b2d50af5 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:12.786072 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-952ec722-837f-485f-a63f-d6c6b2d50af5 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:12.786457 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-952ec722-837f-485f-a63f-d6c6b2d50af5 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:12.793443 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-952ec722-837f-485f-a63f-d6c6b2d50af5 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:12.795078 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-952ec722-837f-485f-a63f-d6c6b2d50af5 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:12.796070 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3be8144-0f44-472e-b8d6-ad035fe7bee3 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:12.807604 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-952ec722-837f-485f-a63f-d6c6b2d50af5 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 21:21:12.807604 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:12.807604 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:12.807604 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:12.807604 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:12.807604 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:12.807604 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:12.807604 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:12.807604 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:12.807604 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:12.807604 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:12.807604 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:12.807604 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 29 21:21:12.807604 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 29 21:21:12.807604 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 29 21:21:12.808568 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 29 21:21:12.808568 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:12.808568 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:12.808568 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:12.808568 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:12.808568 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:12.808568 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:12.808568 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:12.808568 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:12.808568 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:12.808568 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:12.808568 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:12.808568 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:12.808568 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:12.808568 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:12.808568 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:12.808568 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:12.809618 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 29 21:21:12.809618 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 29 21:21:12.809618 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 29 21:21:12.809618 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 29 21:21:12.809618 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 29 21:21:12.809618 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 29 21:21:12.809618 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 21:21:12.809618 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:21:12.809618 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 400/801] 10.222.0.90 () {56 vars in 1111 bytes} [Mon Aug 29 21:21:12 2022] GET /identity/v3/projects/service => generated 88 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:21:12.809618 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-952ec722-837f-485f-a63f-d6c6b2d50af5 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:12.838910 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16dc9790-1624-4c6a-a9c4-edbc32c23b3f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:12.839618 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16dc9790-1624-4c6a-a9c4-edbc32c23b3f None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:12.840038 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16dc9790-1624-4c6a-a9c4-edbc32c23b3f None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:12.840409 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16dc9790-1624-4c6a-a9c4-edbc32c23b3f None admin] PATH_INFO: `/v3/roles/service` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:12.852365 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16dc9790-1624-4c6a-a9c4-edbc32c23b3f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 21:21:12.857419 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-16dc9790-1624-4c6a-a9c4-edbc32c23b3f None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 29 21:21:12.857419 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:12.857419 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:12.857419 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:12.857419 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:12.857419 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:12.857419 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:12.857419 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:12.857419 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:12.857419 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:12.857419 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:12.857419 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:12.857419 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 21:21:12.857419 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 21:21:12.857419 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 21:21:12.857419 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 21:21:12.858637 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 21:21:12.858637 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 21:21:12.858637 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 21:21:12.858637 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 21:21:12.858637 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:12.858637 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:12.858637 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:12.858637 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:12.858637 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:12.858637 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:12.858637 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:12.858637 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:12.858637 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:12.858637 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:12.858637 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:12.858637 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:12.859727 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:12.859727 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:12.859727 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:12.859727 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:12.859727 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 21:21:12.859727 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 21:21:12.859727 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 21:21:12.859727 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 21:21:12.859727 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 21:21:12.859727 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 21:21:12.859727 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 29 21:21:12.859727 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:21:12.861656 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 402/802] 10.222.0.90 () {56 vars in 1102 bytes} [Mon Aug 29 21:21:12 2022] GET /identity/v3/roles/service => generated 85 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:21:12.863050 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16dc9790-1624-4c6a-a9c4-edbc32c23b3f None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:12.891078 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:12.939066 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-952ec722-837f-485f-a63f-d6c6b2d50af5 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:12.975015 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16dc9790-1624-4c6a-a9c4-edbc32c23b3f None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:12.980819 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-436d3ce7-8676-4593-9ed6-f0208d2d8d34 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:12.981463 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-436d3ce7-8676-4593-9ed6-f0208d2d8d34 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:12.981743 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-436d3ce7-8676-4593-9ed6-f0208d2d8d34 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:12.982060 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-436d3ce7-8676-4593-9ed6-f0208d2d8d34 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:12.991808 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-436d3ce7-8676-4593-9ed6-f0208d2d8d34 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:12.993237 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-436d3ce7-8676-4593-9ed6-f0208d2d8d34 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:12.994466 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:13.006300 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-436d3ce7-8676-4593-9ed6-f0208d2d8d34 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 29 21:21:13.006300 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:13.006300 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:13.006300 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:13.006300 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:13.006300 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:13.006300 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:13.006300 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:13.006300 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:13.006300 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:13.006300 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:13.006300 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:13.006300 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 29 21:21:13.006300 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 29 21:21:13.006300 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 29 21:21:13.006300 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 29 21:21:13.007296 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:13.007296 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:13.007296 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 29 21:21:13.007296 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 21:21:13.007296 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 29 21:21:13.007296 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 21:21:13.007296 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:13.007296 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:13.007296 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:13.007296 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:13.007296 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:13.007296 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:13.007296 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:13.007296 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:13.007296 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:13.007296 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:13.007296 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:13.007296 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:13.008905 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:13.008905 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:13.008905 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 29 21:21:13.008905 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 29 21:21:13.008905 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 29 21:21:13.008905 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 29 21:21:13.008905 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 29 21:21:13.008905 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 29 21:21:13.008905 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 29 21:21:13.008905 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:21:13.008905 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 401/803] 10.222.0.90 () {56 vars in 1093 bytes} [Mon Aug 29 21:21:12 2022] GET /identity/v3/users/nova => generated 82 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:21:13.008905 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-436d3ce7-8676-4593-9ed6-f0208d2d8d34 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:13.020480 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9f7f139-6758-4eda-be9c-998d82dad319 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:13.021356 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9f7f139-6758-4eda-be9c-998d82dad319 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:13.022412 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9f7f139-6758-4eda-be9c-998d82dad319 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:13.022899 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9f7f139-6758-4eda-be9c-998d82dad319 None admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:13.023845 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9f7f139-6758-4eda-be9c-998d82dad319 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:13.024423 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9f7f139-6758-4eda-be9c-998d82dad319 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:13.026197 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9f7f139-6758-4eda-be9c-998d82dad319 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:13.040066 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 403/804] 10.222.0.90 () {56 vars in 1148 bytes} [Mon Aug 29 21:21:12 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:13.041335 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9f7f139-6758-4eda-be9c-998d82dad319 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:13.125363 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-436d3ce7-8676-4593-9ed6-f0208d2d8d34 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:13.145178 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:13.154912 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9f7f139-6758-4eda-be9c-998d82dad319 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:13.163487 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-342d42fa-1617-4a05-bcbf-b713b823d1f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:13.164095 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-342d42fa-1617-4a05-bcbf-b713b823d1f6 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:13.164375 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-342d42fa-1617-4a05-bcbf-b713b823d1f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:13.164640 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-342d42fa-1617-4a05-bcbf-b713b823d1f6 None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:13.165297 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-342d42fa-1617-4a05-bcbf-b713b823d1f6 None admin] RBAC: Adding query filter params (name=service) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:13.165596 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-342d42fa-1617-4a05-bcbf-b713b823d1f6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:13.166996 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-342d42fa-1617-4a05-bcbf-b713b823d1f6 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:13.179538 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 402/805] 10.222.0.90 () {56 vars in 1103 bytes} [Mon Aug 29 21:21:13 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:13.179538 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-342d42fa-1617-4a05-bcbf-b713b823d1f6 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:13.195259 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76c1331e-00a5-46f3-8cf4-2ca97ea685e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:13.195887 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c1331e-00a5-46f3-8cf4-2ca97ea685e7 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:13.196167 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c1331e-00a5-46f3-8cf4-2ca97ea685e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:13.196442 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c1331e-00a5-46f3-8cf4-2ca97ea685e7 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:13.197030 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76c1331e-00a5-46f3-8cf4-2ca97ea685e7 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:13.198433 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76c1331e-00a5-46f3-8cf4-2ca97ea685e7 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:13.207940 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-76c1331e-00a5-46f3-8cf4-2ca97ea685e7 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 29 21:21:13.207940 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:13.207940 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 29 21:21:13.207940 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 29 21:21:13.207940 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 29 21:21:13.207940 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 29 21:21:13.207940 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 29 21:21:13.207940 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 21:21:13.207940 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 29 21:21:13.207940 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:21:13.207940 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 29 21:21:13.207940 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:21:13.207940 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:13.207940 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:13.207940 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:13.207940 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:13.207940 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:13.208930 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:13.208930 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:13.208930 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:13.208930 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:13.208930 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:13.208930 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:13.208930 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 29 21:21:13.208930 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 29 21:21:13.208930 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 29 21:21:13.208930 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 29 21:21:13.208930 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:13.208930 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:13.208930 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:13.208930 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:13.208930 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:13.208930 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:13.209807 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:13.209807 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:13.209807 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:13.209807 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:13.209807 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:13.209807 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:13.209807 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:13.209807 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:13.209807 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:13.209807 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:13.209807 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 29 21:21:13.209807 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 21:21:13.209807 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 29 21:21:13.209807 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:21:13.209807 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 404/806] 10.222.0.90 () {56 vars in 1129 bytes} [Mon Aug 29 21:21:13 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:21:13.211201 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76c1331e-00a5-46f3-8cf4-2ca97ea685e7 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:13.255505 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:13.294609 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-342d42fa-1617-4a05-bcbf-b713b823d1f6 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:13.319870 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76c1331e-00a5-46f3-8cf4-2ca97ea685e7 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:13.331358 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf4e8360-f08c-4c31-84f8-424d346ac6c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:13.332007 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf4e8360-f08c-4c31-84f8-424d346ac6c9 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:13.332321 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf4e8360-f08c-4c31-84f8-424d346ac6c9 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:13.332593 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf4e8360-f08c-4c31-84f8-424d346ac6c9 None admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:13.333334 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf4e8360-f08c-4c31-84f8-424d346ac6c9 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:13.333640 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf4e8360-f08c-4c31-84f8-424d346ac6c9 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:13.335123 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf4e8360-f08c-4c31-84f8-424d346ac6c9 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:13.354383 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 403/807] 10.222.0.90 () {56 vars in 1133 bytes} [Mon Aug 29 21:21:13 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:13.358843 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31960442-3d64-4b27-91ec-a68102851de9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:13.359141 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf4e8360-f08c-4c31-84f8-424d346ac6c9 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:13.359517 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31960442-3d64-4b27-91ec-a68102851de9 None admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:13.359803 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31960442-3d64-4b27-91ec-a68102851de9 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:13.360080 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31960442-3d64-4b27-91ec-a68102851de9 None admin] PATH_INFO: `/v3/projects/ec03e112e3d547fbab6faf7b485c3fca/users/139bbbde6d884b42b5440727926e2c58/roles/a6e562cdc96c4be691d638990f8403fd` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:13.362114 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31960442-3d64-4b27-91ec-a68102851de9 None admin] RBAC: Authorizing `identity:create_grant(project_id=ec03e112e3d547fbab6faf7b485c3fca, user_id=139bbbde6d884b42b5440727926e2c58, role_id=a6e562cdc96c4be691d638990f8403fd)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:13.364312 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31960442-3d64-4b27-91ec-a68102851de9 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:13.376632 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 405/808] 10.222.0.90 () {58 vars in 1439 bytes} [Mon Aug 29 21:21:13 2022] PUT /identity/v3/projects/ec03e112e3d547fbab6faf7b485c3fca/users/139bbbde6d884b42b5440727926e2c58/roles/a6e562cdc96c4be691d638990f8403fd => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:21:13.378034 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31960442-3d64-4b27-91ec-a68102851de9 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:13.410590 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:13.487851 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf4e8360-f08c-4c31-84f8-424d346ac6c9 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:13.507730 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31960442-3d64-4b27-91ec-a68102851de9 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:13.509544 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:13.535766 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36cd211d-85a8-43c0-8087-8ffdb7a3ae8d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:13.536429 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36cd211d-85a8-43c0-8087-8ffdb7a3ae8d None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:13.536706 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36cd211d-85a8-43c0-8087-8ffdb7a3ae8d None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:13.536979 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36cd211d-85a8-43c0-8087-8ffdb7a3ae8d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:13.553263 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36cd211d-85a8-43c0-8087-8ffdb7a3ae8d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:13.555381 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36cd211d-85a8-43c0-8087-8ffdb7a3ae8d None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:13.561137 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-317f9f73-716a-42f4-976d-84e8b14e4447 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:13.561765 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-317f9f73-716a-42f4-976d-84e8b14e4447 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:13.562092 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-317f9f73-716a-42f4-976d-84e8b14e4447 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:13.562363 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-317f9f73-716a-42f4-976d-84e8b14e4447 None admin] PATH_INFO: `/v3/services` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:13.563000 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-317f9f73-716a-42f4-976d-84e8b14e4447 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:13.563304 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-317f9f73-716a-42f4-976d-84e8b14e4447 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:13.565579 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-317f9f73-716a-42f4-976d-84e8b14e4447 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:13.571216 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 406/809] 10.222.0.90 () {56 vars in 1124 bytes} [Mon Aug 29 21:21:13 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:13.571795 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-317f9f73-716a-42f4-976d-84e8b14e4447 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:13.579711 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-36cd211d-85a8-43c0-8087-8ffdb7a3ae8d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:21:13.579711 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:13.579711 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:13.579711 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:13.579711 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:13.579711 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:13.579711 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:13.579711 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:13.579711 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:13.579711 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:13.579711 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:13.579711 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:13.579711 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 21:21:13.579711 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 21:21:13.579711 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 21:21:13.579711 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 21:21:13.581129 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:13.581129 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:13.581129 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:13.581129 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:13.581129 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:13.581129 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:13.581129 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:13.581129 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:13.581129 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:13.581129 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:13.581129 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:13.581129 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:13.581129 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:13.581129 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:13.581129 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:13.581129 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:13.581129 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 21:21:13.582576 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 21:21:13.582576 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:21:13.582576 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:21:13.582576 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 404/810] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:21:13 2022] GET /identity/v3/domains/Default => generated 87 bytes in 227 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:21:13.587420 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36cd211d-85a8-43c0-8087-8ffdb7a3ae8d None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:13.710237 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:13.720560 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36cd211d-85a8-43c0-8087-8ffdb7a3ae8d None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:13.757827 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c259477b-a191-4e83-9744-adc2a9868912 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:13.758524 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c259477b-a191-4e83-9744-adc2a9868912 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:13.758800 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c259477b-a191-4e83-9744-adc2a9868912 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:13.759068 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c259477b-a191-4e83-9744-adc2a9868912 None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:13.760483 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c259477b-a191-4e83-9744-adc2a9868912 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:13.760785 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c259477b-a191-4e83-9744-adc2a9868912 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:13.762189 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c259477b-a191-4e83-9744-adc2a9868912 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:13.778984 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 405/811] 10.222.0.90 () {56 vars in 1109 bytes} [Mon Aug 29 21:21:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:21:13.779987 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c259477b-a191-4e83-9744-adc2a9868912 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:13.808800 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-317f9f73-716a-42f4-976d-84e8b14e4447 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:13.852546 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d309a4fd-140e-4660-94f7-cbd0b6f28fec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:13.853542 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d309a4fd-140e-4660-94f7-cbd0b6f28fec None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:13.859400 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d309a4fd-140e-4660-94f7-cbd0b6f28fec None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:13.859982 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d309a4fd-140e-4660-94f7-cbd0b6f28fec None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:13.886476 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c259477b-a191-4e83-9744-adc2a9868912 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:13.908940 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:13.934728 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f8de573-79f9-4e52-a006-f8554759e336 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:13.935722 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f8de573-79f9-4e52-a006-f8554759e336 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:13.936201 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f8de573-79f9-4e52-a006-f8554759e336 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:13.936670 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f8de573-79f9-4e52-a006-f8554759e336 None admin] PATH_INFO: `/v3/services` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:13.937566 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f8de573-79f9-4e52-a006-f8554759e336 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:13.938091 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f8de573-79f9-4e52-a006-f8554759e336 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:13.939745 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f8de573-79f9-4e52-a006-f8554759e336 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:13.945752 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 406/812] 10.222.0.90 () {56 vars in 1124 bytes} [Mon Aug 29 21:21:13 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:13.946816 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f8de573-79f9-4e52-a006-f8554759e336 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:13.962033 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d309a4fd-140e-4660-94f7-cbd0b6f28fec None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:13.963777 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d309a4fd-140e-4660-94f7-cbd0b6f28fec None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:13.965149 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:14.056351 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f8de573-79f9-4e52-a006-f8554759e336 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:14.085077 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 407/813] 10.222.0.90 () {58 vars in 1282 bytes} [Mon Aug 29 21:21:13 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 513 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 29 21:21:14.085854 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d309a4fd-140e-4660-94f7-cbd0b6f28fec None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:14.097970 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-048be3bb-4cf5-443a-b459-9c87f3abe0a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:14.098610 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-048be3bb-4cf5-443a-b459-9c87f3abe0a5 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:14.098886 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-048be3bb-4cf5-443a-b459-9c87f3abe0a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:14.099150 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-048be3bb-4cf5-443a-b459-9c87f3abe0a5 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:14.107070 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-048be3bb-4cf5-443a-b459-9c87f3abe0a5 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:14.108590 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-048be3bb-4cf5-443a-b459-9c87f3abe0a5 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:14.121272 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-048be3bb-4cf5-443a-b459-9c87f3abe0a5 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 29 21:21:14.121272 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:14.121272 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:14.121272 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:14.121272 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:14.121272 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:14.121272 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:14.121272 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:14.121272 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:14.121272 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:14.121272 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:14.121272 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:14.121272 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 29 21:21:14.121272 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 29 21:21:14.121272 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 29 21:21:14.121272 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 29 21:21:14.122466 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:14.122466 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:14.122466 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 29 21:21:14.122466 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 21:21:14.122466 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 29 21:21:14.122466 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 21:21:14.122466 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:14.122466 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:14.122466 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:14.122466 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:14.122466 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:14.122466 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:14.122466 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:14.122466 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:14.122466 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:14.122466 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:14.122466 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:14.122466 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:14.123704 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:14.123704 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:14.123704 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 29 21:21:14.123704 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 29 21:21:14.123704 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 29 21:21:14.123704 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 29 21:21:14.123704 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 29 21:21:14.123704 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 29 21:21:14.123704 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 29 21:21:14.123704 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:21:14.123704 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 407/814] 10.222.0.90 () {56 vars in 1117 bytes} [Mon Aug 29 21:21:13 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:21:14.124309 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2556f315-83c5-4ab2-9229-791cb74566ff None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:14.124309 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2556f315-83c5-4ab2-9229-791cb74566ff None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:14.124445 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2556f315-83c5-4ab2-9229-791cb74566ff None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:14.130213 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 408/815] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:21:14 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:21:14.130213 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2556f315-83c5-4ab2-9229-791cb74566ff None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:14.163599 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:14.200839 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d309a4fd-140e-4660-94f7-cbd0b6f28fec None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:14.210397 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:14.237375 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b48b6ee-4c3f-4cd7-a830-4c2ab42aedd8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:14.238016 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b48b6ee-4c3f-4cd7-a830-4c2ab42aedd8 None admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:14.238350 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b48b6ee-4c3f-4cd7-a830-4c2ab42aedd8 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:14.238560 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b48b6ee-4c3f-4cd7-a830-4c2ab42aedd8 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:14.238995 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2556f315-83c5-4ab2-9229-791cb74566ff None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:14.239223 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b48b6ee-4c3f-4cd7-a830-4c2ab42aedd8 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:14.240576 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b48b6ee-4c3f-4cd7-a830-4c2ab42aedd8 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:14.265203 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 408/816] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:21:14 2022] POST /identity/v3/endpoints => generated 346 bytes in 180 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:21:14.266471 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe90858b-3f91-493b-8cda-310ec5a2a47f None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:14.266760 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe90858b-3f91-493b-8cda-310ec5a2a47f None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:14.267034 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe90858b-3f91-493b-8cda-310ec5a2a47f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:14.287914 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-fe90858b-3f91-493b-8cda-310ec5a2a47f None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:21:14.289176 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-fe90858b-3f91-493b-8cda-310ec5a2a47f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:21:14.292884 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1002df9a-4099-4acb-bb99-1e05829a3211 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:14.293641 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1002df9a-4099-4acb-bb99-1e05829a3211 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:14.294065 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1002df9a-4099-4acb-bb99-1e05829a3211 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:14.294724 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1002df9a-4099-4acb-bb99-1e05829a3211 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:14.310079 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1002df9a-4099-4acb-bb99-1e05829a3211 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:14.311917 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1002df9a-4099-4acb-bb99-1e05829a3211 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:14.330463 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-1002df9a-4099-4acb-bb99-1e05829a3211 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:21:14.330463 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:14.330463 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:14.330463 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:14.330463 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:14.330463 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:14.330463 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:14.330463 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:14.330463 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:14.330463 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:14.330463 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:14.330463 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:14.330463 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 21:21:14.330463 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 21:21:14.330463 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 21:21:14.330463 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 21:21:14.331522 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:14.331522 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:14.331522 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:14.331522 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:14.331522 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:14.331522 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:14.331522 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:14.331522 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:14.331522 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:14.331522 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:14.331522 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:14.331522 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:14.331522 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:14.331522 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:14.331522 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:14.331522 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:14.331522 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 21:21:14.332440 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 21:21:14.332440 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:21:14.332440 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:21:14.334948 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 409/817] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:21:14 2022] GET /identity/v3/domains/Default => generated 87 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:21:14.336128 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1002df9a-4099-4acb-bb99-1e05829a3211 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:14.389633 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 409/818] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:21:14 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 21:21:14.391035 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe90858b-3f91-493b-8cda-310ec5a2a47f None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:14.474378 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1002df9a-4099-4acb-bb99-1e05829a3211 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:14.495788 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:14.503027 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:14.516357 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-214ffcd9-244c-4a27-8629-7ee215168689 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:14.517909 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-214ffcd9-244c-4a27-8629-7ee215168689 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:14.518398 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-214ffcd9-244c-4a27-8629-7ee215168689 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:14.518697 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-214ffcd9-244c-4a27-8629-7ee215168689 None admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:14.519840 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-214ffcd9-244c-4a27-8629-7ee215168689 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:14.520175 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-214ffcd9-244c-4a27-8629-7ee215168689 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:14.522572 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-214ffcd9-244c-4a27-8629-7ee215168689 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:14.545941 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 410/819] 10.222.0.90 () {56 vars in 1149 bytes} [Mon Aug 29 21:21:14 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:14.551471 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-214ffcd9-244c-4a27-8629-7ee215168689 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:14.568178 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe90858b-3f91-493b-8cda-310ec5a2a47f None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:14.642445 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90a16541-9ffa-4bf2-b6be-8b3959660aa8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:14.643903 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a16541-9ffa-4bf2-b6be-8b3959660aa8 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:14.644740 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a16541-9ffa-4bf2-b6be-8b3959660aa8 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:14.645513 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a16541-9ffa-4bf2-b6be-8b3959660aa8 None admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:14.646556 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90a16541-9ffa-4bf2-b6be-8b3959660aa8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:14.647071 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90a16541-9ffa-4bf2-b6be-8b3959660aa8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:14.649207 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90a16541-9ffa-4bf2-b6be-8b3959660aa8 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:14.678160 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 410/820] 10.222.0.90 () {56 vars in 1109 bytes} [Mon Aug 29 21:21:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 283 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:14.681845 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90a16541-9ffa-4bf2-b6be-8b3959660aa8 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:14.743753 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-214ffcd9-244c-4a27-8629-7ee215168689 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:14.802083 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90a16541-9ffa-4bf2-b6be-8b3959660aa8 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:14.807015 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ff6d4e7-695d-44a5-9456-23b62e3dbc72 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:14.807666 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ff6d4e7-695d-44a5-9456-23b62e3dbc72 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:14.807950 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ff6d4e7-695d-44a5-9456-23b62e3dbc72 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:14.808230 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ff6d4e7-695d-44a5-9456-23b62e3dbc72 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:14.818966 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:14.823622 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ff6d4e7-695d-44a5-9456-23b62e3dbc72 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:14.826269 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ff6d4e7-695d-44a5-9456-23b62e3dbc72 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:14.845012 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b381b93e-9b13-4d4a-93c3-0b81eef84433 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:14.846672 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b381b93e-9b13-4d4a-93c3-0b81eef84433 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:14.847116 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b381b93e-9b13-4d4a-93c3-0b81eef84433 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:14.847559 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b381b93e-9b13-4d4a-93c3-0b81eef84433 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:14.850238 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-6ff6d4e7-695d-44a5-9456-23b62e3dbc72 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:21:14.850238 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:14.850238 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:14.850238 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:14.850238 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:14.850238 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:14.850238 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:14.850238 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:14.850238 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:14.850238 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:14.850238 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:14.850238 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:14.850238 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 21:21:14.850238 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 21:21:14.850238 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 21:21:14.850238 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 21:21:14.851420 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:14.851420 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:14.851420 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:14.851420 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:14.851420 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:14.851420 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:14.851420 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:14.851420 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:14.851420 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:14.851420 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:14.851420 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:14.851420 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:14.851420 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:14.851420 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:14.851420 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:14.851420 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:14.851420 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 21:21:14.852619 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 21:21:14.852619 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:21:14.852619 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:21:14.854852 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 411/821] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:21:14 2022] GET /identity/v3/domains/Default => generated 87 bytes in 304 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 29 21:21:14.860265 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ff6d4e7-695d-44a5-9456-23b62e3dbc72 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:14.862643 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b381b93e-9b13-4d4a-93c3-0b81eef84433 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:14.865155 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b381b93e-9b13-4d4a-93c3-0b81eef84433 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:14.879009 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-b381b93e-9b13-4d4a-93c3-0b81eef84433 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 21:21:14.879009 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:14.879009 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:14.879009 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:14.879009 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:14.879009 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:14.879009 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:14.879009 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:14.879009 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:14.879009 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:14.879009 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:14.879009 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:14.879009 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 29 21:21:14.879009 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 29 21:21:14.879009 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 29 21:21:14.880046 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 29 21:21:14.880046 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:14.880046 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:14.880046 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:14.880046 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:14.880046 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:14.880046 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:14.880046 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:14.880046 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:14.880046 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:14.880046 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:14.880046 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:14.880046 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:14.880046 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:14.880046 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:14.880046 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:14.880046 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:14.880991 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 29 21:21:14.880991 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 29 21:21:14.880991 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 29 21:21:14.880991 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 29 21:21:14.880991 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 29 21:21:14.880991 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 29 21:21:14.880991 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 21:21:14.880991 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:21:14.883764 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 411/822] 10.222.0.90 () {56 vars in 1111 bytes} [Mon Aug 29 21:21:14 2022] GET /identity/v3/projects/service => generated 88 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:21:14.885005 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b381b93e-9b13-4d4a-93c3-0b81eef84433 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:15.009628 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b381b93e-9b13-4d4a-93c3-0b81eef84433 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:15.009865 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:15.049076 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cd7efb1-a456-443d-aa78-d3f2fdae2025 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:15.049970 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cd7efb1-a456-443d-aa78-d3f2fdae2025 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:15.050860 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cd7efb1-a456-443d-aa78-d3f2fdae2025 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:15.051249 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cd7efb1-a456-443d-aa78-d3f2fdae2025 None admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:15.052175 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cd7efb1-a456-443d-aa78-d3f2fdae2025 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:15.052760 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cd7efb1-a456-443d-aa78-d3f2fdae2025 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:15.054493 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cd7efb1-a456-443d-aa78-d3f2fdae2025 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:15.073953 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 412/823] 10.222.0.90 () {56 vars in 1109 bytes} [Mon Aug 29 21:21:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:15.075088 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cd7efb1-a456-443d-aa78-d3f2fdae2025 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:15.089117 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ff6d4e7-695d-44a5-9456-23b62e3dbc72 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:15.118474 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:15.128841 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b750445e-a825-411e-bae8-619568e8b854 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:15.129747 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b750445e-a825-411e-bae8-619568e8b854 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:15.130229 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b750445e-a825-411e-bae8-619568e8b854 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:15.131080 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b750445e-a825-411e-bae8-619568e8b854 None admin] PATH_INFO: `/v3/services/network` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:15.131858 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b750445e-a825-411e-bae8-619568e8b854 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:15.133442 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b750445e-a825-411e-bae8-619568e8b854 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:15.143917 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-b750445e-a825-411e-bae8-619568e8b854 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Aug 29 21:21:15.143917 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:15.143917 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 29 21:21:15.143917 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 29 21:21:15.143917 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 29 21:21:15.143917 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 29 21:21:15.143917 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 29 21:21:15.143917 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 21:21:15.143917 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Aug 29 21:21:15.143917 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:21:15.143917 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 29 21:21:15.143917 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:21:15.143917 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:15.143917 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:15.143917 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:15.143917 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:15.143917 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:15.144981 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:15.144981 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:15.144981 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:15.144981 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:15.144981 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:15.144981 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:15.144981 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 29 21:21:15.144981 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 29 21:21:15.144981 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 29 21:21:15.144981 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 29 21:21:15.144981 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:15.144981 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:15.144981 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:15.144981 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:15.144981 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:15.144981 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:15.145889 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:15.145889 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:15.145889 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:15.145889 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:15.145889 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:15.145889 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:15.145889 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:15.145889 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:15.145889 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:15.145889 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:15.145889 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 29 21:21:15.145889 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 21:21:15.145889 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Aug 29 21:21:15.145889 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:21:15.153453 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 412/824] 10.222.0.90 () {56 vars in 1111 bytes} [Mon Aug 29 21:21:14 2022] GET /identity/v3/services/network => generated 88 bytes in 293 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:21:15.154634 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b750445e-a825-411e-bae8-619568e8b854 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:15.219933 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cd7efb1-a456-443d-aa78-d3f2fdae2025 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:15.267135 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:15.275272 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b750445e-a825-411e-bae8-619568e8b854 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:15.275681 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adc22d6c-5dfe-4742-8b10-27d8cb39ee54 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:15.276306 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adc22d6c-5dfe-4742-8b10-27d8cb39ee54 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:15.276590 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adc22d6c-5dfe-4742-8b10-27d8cb39ee54 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:15.276865 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adc22d6c-5dfe-4742-8b10-27d8cb39ee54 None admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:15.277566 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adc22d6c-5dfe-4742-8b10-27d8cb39ee54 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:15.277913 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adc22d6c-5dfe-4742-8b10-27d8cb39ee54 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:15.280567 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adc22d6c-5dfe-4742-8b10-27d8cb39ee54 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:15.297327 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 413/825] 10.222.0.90 () {56 vars in 1148 bytes} [Mon Aug 29 21:21:15 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:15.298298 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adc22d6c-5dfe-4742-8b10-27d8cb39ee54 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:15.315922 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-147a5754-50c6-425f-bd96-8760734f829e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:15.317072 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-147a5754-50c6-425f-bd96-8760734f829e None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:15.317560 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-147a5754-50c6-425f-bd96-8760734f829e None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:15.318124 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-147a5754-50c6-425f-bd96-8760734f829e None admin] PATH_INFO: `/v3/projects/service` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:15.325519 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-147a5754-50c6-425f-bd96-8760734f829e None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:15.327349 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-147a5754-50c6-425f-bd96-8760734f829e None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:15.340451 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-147a5754-50c6-425f-bd96-8760734f829e None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 21:21:15.340451 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:15.340451 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:15.340451 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:15.340451 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:15.340451 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:15.340451 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:15.340451 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:15.340451 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:15.340451 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:15.340451 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:15.340451 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:15.340451 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 29 21:21:15.340451 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 29 21:21:15.340451 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 29 21:21:15.341419 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 29 21:21:15.341419 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:15.341419 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:15.341419 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:15.341419 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:15.341419 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:15.341419 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:15.341419 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:15.341419 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:15.341419 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:15.341419 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:15.341419 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:15.341419 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:15.341419 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:15.341419 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:15.341419 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:15.341419 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:15.342369 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 29 21:21:15.342369 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 29 21:21:15.342369 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 29 21:21:15.342369 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 29 21:21:15.342369 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 29 21:21:15.342369 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 29 21:21:15.342369 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 21:21:15.342369 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:21:15.344802 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 413/826] 10.222.0.90 () {56 vars in 1111 bytes} [Mon Aug 29 21:21:15 2022] GET /identity/v3/projects/service => generated 88 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:21:15.346853 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-147a5754-50c6-425f-bd96-8760734f829e None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:15.428662 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:15.441340 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adc22d6c-5dfe-4742-8b10-27d8cb39ee54 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:15.472438 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-147a5754-50c6-425f-bd96-8760734f829e None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:15.481294 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16b9b7d0-8175-4744-a3c0-f06964be0068 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:15.481960 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b9b7d0-8175-4744-a3c0-f06964be0068 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:15.482253 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b9b7d0-8175-4744-a3c0-f06964be0068 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:15.482588 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b9b7d0-8175-4744-a3c0-f06964be0068 None admin] PATH_INFO: `/v3/services` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:15.483428 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16b9b7d0-8175-4744-a3c0-f06964be0068 None admin] RBAC: Adding query filter params (name=network) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:15.483781 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16b9b7d0-8175-4744-a3c0-f06964be0068 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:15.485158 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16b9b7d0-8175-4744-a3c0-f06964be0068 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:15.490492 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 414/827] 10.222.0.90 () {56 vars in 1112 bytes} [Mon Aug 29 21:21:15 2022] GET /identity/v3/services?name=network => generated 124 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:15.491304 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16b9b7d0-8175-4744-a3c0-f06964be0068 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:15.514625 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ac029c7-7b93-4770-98ca-90e140d026fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:15.515871 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ac029c7-7b93-4770-98ca-90e140d026fe None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:15.516437 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ac029c7-7b93-4770-98ca-90e140d026fe None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:15.517202 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ac029c7-7b93-4770-98ca-90e140d026fe None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:15.518774 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ac029c7-7b93-4770-98ca-90e140d026fe None admin] RBAC: Adding query filter params (role.id=a6e562cdc96c4be691d638990f8403fd, scope.project.id=ec03e112e3d547fbab6faf7b485c3fca, user.id=a35b5b7dea7b421da056f063d2436ffb) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:15.519458 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ac029c7-7b93-4770-98ca-90e140d026fe None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:15.521078 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ac029c7-7b93-4770-98ca-90e140d026fe None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:15.528005 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 414/828] 10.222.0.90 () {56 vars in 1374 bytes} [Mon Aug 29 21:21:15 2022] GET /identity/v3/role_assignments?user.id=a35b5b7dea7b421da056f063d2436ffb&scope.project.id=ec03e112e3d547fbab6faf7b485c3fca&role.id=a6e562cdc96c4be691d638990f8403fd => generated 608 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:15.529519 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55949792-dcb3-4405-b535-07d6d84fa19b None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:15.529818 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55949792-dcb3-4405-b535-07d6d84fa19b None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:15.530098 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55949792-dcb3-4405-b535-07d6d84fa19b None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:15.531448 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 415/829] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:21:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:21:15.531990 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55949792-dcb3-4405-b535-07d6d84fa19b None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:15.577760 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:15.627612 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16b9b7d0-8175-4744-a3c0-f06964be0068 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:15.672009 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55949792-dcb3-4405-b535-07d6d84fa19b None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:15.677694 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b2c4887-2baf-4fc7-aaad-3c1ed16b6c3a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:15.677887 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b2c4887-2baf-4fc7-aaad-3c1ed16b6c3a None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:15.678111 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b2c4887-2baf-4fc7-aaad-3c1ed16b6c3a None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:15.678328 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b2c4887-2baf-4fc7-aaad-3c1ed16b6c3a None admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:15.679033 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b2c4887-2baf-4fc7-aaad-3c1ed16b6c3a None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:15.679339 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b2c4887-2baf-4fc7-aaad-3c1ed16b6c3a None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:15.680775 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b2c4887-2baf-4fc7-aaad-3c1ed16b6c3a None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:15.702869 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 415/830] 10.222.0.90 () {56 vars in 1148 bytes} [Mon Aug 29 21:21:15 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:15.702869 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42f7c0de-1e0c-465f-8e63-9bd5613f3c98 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:15.702869 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42f7c0de-1e0c-465f-8e63-9bd5613f3c98 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:15.702869 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42f7c0de-1e0c-465f-8e63-9bd5613f3c98 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:15.720149 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-42f7c0de-1e0c-465f-8e63-9bd5613f3c98 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:21:15.722541 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-42f7c0de-1e0c-465f-8e63-9bd5613f3c98 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:21:15.726726 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fac76f5-a13b-4069-80b2-2b799748614a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:15.727352 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fac76f5-a13b-4069-80b2-2b799748614a None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:15.727634 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fac76f5-a13b-4069-80b2-2b799748614a None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:15.727909 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fac76f5-a13b-4069-80b2-2b799748614a None admin] PATH_INFO: `/v3/services` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:15.728551 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fac76f5-a13b-4069-80b2-2b799748614a None admin] RBAC: Adding query filter params (type=network) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:15.728858 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fac76f5-a13b-4069-80b2-2b799748614a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:15.730491 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fac76f5-a13b-4069-80b2-2b799748614a None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:15.735006 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:15.738724 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 416/831] 10.222.0.90 () {56 vars in 1112 bytes} [Mon Aug 29 21:21:15 2022] GET /identity/v3/services?type=network => generated 350 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:15.739663 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fac76f5-a13b-4069-80b2-2b799748614a None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:15.807839 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 416/832] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:21:15 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 21:21:15.813812 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42f7c0de-1e0c-465f-8e63-9bd5613f3c98 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:15.879662 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:15.897093 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fac76f5-a13b-4069-80b2-2b799748614a None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:15.935626 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a316eb9b-5af5-4fde-b0c9-9a92cb08e4db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:15.936598 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a316eb9b-5af5-4fde-b0c9-9a92cb08e4db None admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:15.937381 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a316eb9b-5af5-4fde-b0c9-9a92cb08e4db None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:15.937848 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a316eb9b-5af5-4fde-b0c9-9a92cb08e4db None admin] PATH_INFO: `/v3/projects/ec03e112e3d547fbab6faf7b485c3fca/users/0c0b0da87b124de1be00faec4bbdb404/roles/a94bdb4c5796428baf63644c062dfcb2` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:15.959760 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a316eb9b-5af5-4fde-b0c9-9a92cb08e4db None admin] RBAC: Authorizing `identity:create_grant(project_id=ec03e112e3d547fbab6faf7b485c3fca, user_id=0c0b0da87b124de1be00faec4bbdb404, role_id=a94bdb4c5796428baf63644c062dfcb2)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:15.966672 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a316eb9b-5af5-4fde-b0c9-9a92cb08e4db None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:15.988004 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 417/833] 10.222.0.90 () {58 vars in 1439 bytes} [Mon Aug 29 21:21:15 2022] PUT /identity/v3/projects/ec03e112e3d547fbab6faf7b485c3fca/users/0c0b0da87b124de1be00faec4bbdb404/roles/a94bdb4c5796428baf63644c062dfcb2 => generated 0 bytes in 249 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:21:16.011884 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42f7c0de-1e0c-465f-8e63-9bd5613f3c98 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:16.055249 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09c25918-3adc-4709-af41-9ba639ec159a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:16.056135 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c25918-3adc-4709-af41-9ba639ec159a None admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:16.056593 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c25918-3adc-4709-af41-9ba639ec159a None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:16.057030 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c25918-3adc-4709-af41-9ba639ec159a None admin] PATH_INFO: `/v3/endpoints` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:16.057957 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09c25918-3adc-4709-af41-9ba639ec159a None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:16.059586 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09c25918-3adc-4709-af41-9ba639ec159a None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:16.084185 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 417/834] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:21:15 2022] POST /identity/v3/endpoints => generated 337 bytes in 276 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:21:16.163383 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a316eb9b-5af5-4fde-b0c9-9a92cb08e4db None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:16.280123 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a316eb9b-5af5-4fde-b0c9-9a92cb08e4db None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:16.322038 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:16.333012 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03870b75-800e-46f5-ad40-17c68f6dd14a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:16.333670 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03870b75-800e-46f5-ad40-17c68f6dd14a None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:16.333991 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03870b75-800e-46f5-ad40-17c68f6dd14a None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:16.334264 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03870b75-800e-46f5-ad40-17c68f6dd14a None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:16.346429 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03870b75-800e-46f5-ad40-17c68f6dd14a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 21:21:16.351330 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-03870b75-800e-46f5-ad40-17c68f6dd14a None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 21:21:16.351330 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:16.351330 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:16.351330 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:16.351330 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:16.351330 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:16.351330 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:16.351330 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:16.351330 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:16.351330 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:16.351330 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:16.351330 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:16.351330 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 21:21:16.351330 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 21:21:16.351330 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 21:21:16.351330 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 21:21:16.352301 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 21:21:16.352301 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 21:21:16.352301 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 21:21:16.352301 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 21:21:16.352301 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:16.352301 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:16.352301 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:16.352301 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:16.352301 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:16.352301 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:16.352301 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:16.352301 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:16.352301 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:16.352301 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:16.352301 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:16.352301 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:16.353176 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:16.353176 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:16.353176 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:16.353176 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:16.353176 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 21:21:16.353176 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 21:21:16.353176 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 21:21:16.353176 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 21:21:16.353176 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 21:21:16.353176 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 21:21:16.353176 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 21:21:16.353176 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:21:16.353176 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 418/835] 10.222.0.90 () {56 vars in 1096 bytes} [Mon Aug 29 21:21:16 2022] GET /identity/v3/roles/admin => generated 83 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:21:16.358748 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09c25918-3adc-4709-af41-9ba639ec159a None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:16.362127 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:16.427695 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2f2bd83-ccb4-4169-87e8-b106911ae1e0 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:16.428339 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2f2bd83-ccb4-4169-87e8-b106911ae1e0 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:16.429394 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2f2bd83-ccb4-4169-87e8-b106911ae1e0 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:16.431776 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 419/836] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:21:16 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 29 21:21:16.436928 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f48655c-2167-4e47-a20c-c177ffd85f5e None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:16.437385 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f48655c-2167-4e47-a20c-c177ffd85f5e None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:16.437851 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f48655c-2167-4e47-a20c-c177ffd85f5e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:16.475236 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-9f48655c-2167-4e47-a20c-c177ffd85f5e None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:21:16.477238 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-9f48655c-2167-4e47-a20c-c177ffd85f5e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:21:16.483494 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09c25918-3adc-4709-af41-9ba639ec159a None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:16.531483 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7ad0868-9c65-4e51-83df-ee31e3b8db9e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:16.532489 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7ad0868-9c65-4e51-83df-ee31e3b8db9e None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:16.533007 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7ad0868-9c65-4e51-83df-ee31e3b8db9e None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:16.534077 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7ad0868-9c65-4e51-83df-ee31e3b8db9e None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:16.534963 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7ad0868-9c65-4e51-83df-ee31e3b8db9e None admin] RBAC: Adding query filter params (name=admin) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:16.535482 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7ad0868-9c65-4e51-83df-ee31e3b8db9e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:16.537510 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7ad0868-9c65-4e51-83df-ee31e3b8db9e None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:16.550297 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 418/837] 10.222.0.90 () {56 vars in 1099 bytes} [Mon Aug 29 21:21:16 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:16.555959 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7ad0868-9c65-4e51-83df-ee31e3b8db9e None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:16.581463 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 420/838] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:21:16 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 21:21:16.634201 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:16.674286 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7ad0868-9c65-4e51-83df-ee31e3b8db9e None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:16.709576 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f90c2df6-8f07-42e3-8499-881fadba1064 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:16.710559 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f90c2df6-8f07-42e3-8499-881fadba1064 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:16.711010 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f90c2df6-8f07-42e3-8499-881fadba1064 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:16.711433 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f90c2df6-8f07-42e3-8499-881fadba1064 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:16.723278 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f90c2df6-8f07-42e3-8499-881fadba1064 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:16.724971 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f90c2df6-8f07-42e3-8499-881fadba1064 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:16.740727 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-f90c2df6-8f07-42e3-8499-881fadba1064 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:21:16.740727 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:16.740727 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:16.740727 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:16.740727 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:16.740727 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:16.740727 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:16.740727 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:16.740727 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:16.740727 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:16.740727 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:16.740727 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:16.740727 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 21:21:16.740727 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 21:21:16.740727 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 21:21:16.740727 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 21:21:16.741956 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:16.741956 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:16.741956 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:16.741956 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:16.741956 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:16.741956 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:16.741956 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:16.741956 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:16.741956 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:16.741956 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:16.741956 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:16.741956 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:16.741956 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:16.741956 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:16.741956 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:16.741956 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:16.741956 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 21:21:16.743126 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 21:21:16.743126 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:21:16.743126 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:21:16.744931 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 419/839] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:21:16 2022] GET /identity/v3/domains/Default => generated 87 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:21:16.750376 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f48655c-2167-4e47-a20c-c177ffd85f5e None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:16.871703 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f48655c-2167-4e47-a20c-c177ffd85f5e None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:16.872827 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f90c2df6-8f07-42e3-8499-881fadba1064 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:16.895310 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:16.911377 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae844f8c-c51c-4cbc-b0ec-f8f08816e791 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:16.912261 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae844f8c-c51c-4cbc-b0ec-f8f08816e791 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:16.912375 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae844f8c-c51c-4cbc-b0ec-f8f08816e791 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:16.912631 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae844f8c-c51c-4cbc-b0ec-f8f08816e791 None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:16.913308 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae844f8c-c51c-4cbc-b0ec-f8f08816e791 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:16.913584 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae844f8c-c51c-4cbc-b0ec-f8f08816e791 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:16.914972 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae844f8c-c51c-4cbc-b0ec-f8f08816e791 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:16.928565 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 421/840] 10.222.0.90 () {56 vars in 1109 bytes} [Mon Aug 29 21:21:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:16.934094 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae844f8c-c51c-4cbc-b0ec-f8f08816e791 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:16.989604 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f90c2df6-8f07-42e3-8499-881fadba1064 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:17.007494 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:17.027077 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86860cf0-c97f-43a4-a697-d248bab5ade0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:17.028222 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86860cf0-c97f-43a4-a697-d248bab5ade0 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:17.028735 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86860cf0-c97f-43a4-a697-d248bab5ade0 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:17.029285 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86860cf0-c97f-43a4-a697-d248bab5ade0 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:17.030173 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86860cf0-c97f-43a4-a697-d248bab5ade0 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:17.031947 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86860cf0-c97f-43a4-a697-d248bab5ade0 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:17.042615 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-86860cf0-c97f-43a4-a697-d248bab5ade0 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 29 21:21:17.042615 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:17.042615 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 29 21:21:17.042615 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 29 21:21:17.042615 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 29 21:21:17.042615 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 29 21:21:17.042615 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 29 21:21:17.042615 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 21:21:17.042615 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 29 21:21:17.042615 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:21:17.042615 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 29 21:21:17.042615 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:21:17.042615 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:17.042615 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:17.042615 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:17.042615 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:17.042615 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:17.043630 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:17.043630 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:17.043630 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:17.043630 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:17.043630 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:17.043630 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:17.043630 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 29 21:21:17.043630 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 29 21:21:17.043630 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 29 21:21:17.043630 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 29 21:21:17.043630 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:17.043630 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:17.043630 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:17.043630 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:17.043630 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:17.043630 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:17.044587 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:17.044587 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:17.044587 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:17.044587 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:17.044587 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:17.044587 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:17.044587 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:17.044587 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:17.044587 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:17.044587 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:17.044587 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 29 21:21:17.044587 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 21:21:17.044587 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 29 21:21:17.044587 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:21:17.052462 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae844f8c-c51c-4cbc-b0ec-f8f08816e791 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:17.052791 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 420/841] 10.222.0.90 () {56 vars in 1114 bytes} [Mon Aug 29 21:21:16 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:21:17.054353 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86860cf0-c97f-43a4-a697-d248bab5ade0 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:17.084218 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a1c572b-2b52-426f-92b4-848f9e3dafb7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:17.084768 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a1c572b-2b52-426f-92b4-848f9e3dafb7 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:17.085047 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a1c572b-2b52-426f-92b4-848f9e3dafb7 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:17.085313 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a1c572b-2b52-426f-92b4-848f9e3dafb7 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:17.093304 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a1c572b-2b52-426f-92b4-848f9e3dafb7 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:17.094848 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a1c572b-2b52-426f-92b4-848f9e3dafb7 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:17.107413 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-0a1c572b-2b52-426f-92b4-848f9e3dafb7 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Aug 29 21:21:17.107413 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:17.107413 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:17.107413 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:17.107413 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:17.107413 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:17.107413 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:17.107413 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:17.107413 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:17.107413 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:17.107413 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:17.107413 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:17.107413 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 29 21:21:17.107413 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 29 21:21:17.107413 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 29 21:21:17.107413 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 29 21:21:17.109117 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:17.109117 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:17.109117 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 29 21:21:17.109117 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 21:21:17.109117 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 29 21:21:17.109117 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 21:21:17.109117 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:17.109117 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:17.109117 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:17.109117 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:17.109117 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:17.109117 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:17.109117 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:17.109117 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:17.109117 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:17.109117 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:17.109117 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:17.109117 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:17.110441 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:17.110441 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:17.110441 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 29 21:21:17.110441 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 29 21:21:17.110441 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 29 21:21:17.110441 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 29 21:21:17.110441 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 29 21:21:17.110441 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 29 21:21:17.110441 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Aug 29 21:21:17.110441 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:21:17.110441 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 422/842] 10.222.0.90 () {56 vars in 1096 bytes} [Mon Aug 29 21:21:16 2022] GET /identity/v3/users/swift => generated 83 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:21:17.115284 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a1c572b-2b52-426f-92b4-848f9e3dafb7 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:17.160958 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:17.179850 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86860cf0-c97f-43a4-a697-d248bab5ade0 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:17.218046 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f6474b8-1e72-49ee-986f-be8794a5fb41 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:17.219106 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f6474b8-1e72-49ee-986f-be8794a5fb41 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:17.219602 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f6474b8-1e72-49ee-986f-be8794a5fb41 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:17.220090 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f6474b8-1e72-49ee-986f-be8794a5fb41 None admin] PATH_INFO: `/v3/services` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:17.222498 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f6474b8-1e72-49ee-986f-be8794a5fb41 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:17.223010 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f6474b8-1e72-49ee-986f-be8794a5fb41 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:17.224572 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f6474b8-1e72-49ee-986f-be8794a5fb41 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:17.224974 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a1c572b-2b52-426f-92b4-848f9e3dafb7 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:17.235090 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 421/843] 10.222.0.90 () {56 vars in 1114 bytes} [Mon Aug 29 21:21:17 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:21:17.237427 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f6474b8-1e72-49ee-986f-be8794a5fb41 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:17.262099 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1724dcd-277a-4e84-baba-614a00f391cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:17.262466 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1724dcd-277a-4e84-baba-614a00f391cf None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:17.262753 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1724dcd-277a-4e84-baba-614a00f391cf None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:17.263026 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1724dcd-277a-4e84-baba-614a00f391cf None admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:17.263764 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1724dcd-277a-4e84-baba-614a00f391cf None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:17.264072 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1724dcd-277a-4e84-baba-614a00f391cf None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:17.265502 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1724dcd-277a-4e84-baba-614a00f391cf None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:17.286325 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 423/844] 10.222.0.90 () {56 vars in 1135 bytes} [Mon Aug 29 21:21:17 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:21:17.293842 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1724dcd-277a-4e84-baba-614a00f391cf None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:17.295448 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:17.363952 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f6474b8-1e72-49ee-986f-be8794a5fb41 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:17.401709 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:17.404397 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e640ff1-eb00-4c15-ae9a-13ad7d7c68fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:17.405449 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e640ff1-eb00-4c15-ae9a-13ad7d7c68fe None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:17.405736 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e640ff1-eb00-4c15-ae9a-13ad7d7c68fe None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:17.406045 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e640ff1-eb00-4c15-ae9a-13ad7d7c68fe None admin] PATH_INFO: `/v3/services` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:17.406691 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e640ff1-eb00-4c15-ae9a-13ad7d7c68fe None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:17.406996 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e640ff1-eb00-4c15-ae9a-13ad7d7c68fe None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:17.408748 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1724dcd-277a-4e84-baba-614a00f391cf None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:17.409751 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e640ff1-eb00-4c15-ae9a-13ad7d7c68fe None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:17.421942 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 422/845] 10.222.0.90 () {56 vars in 1114 bytes} [Mon Aug 29 21:21:17 2022] GET /identity/v3/services?type=volumev3 => generated 125 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:17.448829 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-894b8a9c-dd44-4c71-9dbd-a76427dbd05a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:17.449801 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-894b8a9c-dd44-4c71-9dbd-a76427dbd05a None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:17.450291 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-894b8a9c-dd44-4c71-9dbd-a76427dbd05a None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:17.450741 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-894b8a9c-dd44-4c71-9dbd-a76427dbd05a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:17.529302 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-894b8a9c-dd44-4c71-9dbd-a76427dbd05a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:17.531125 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-894b8a9c-dd44-4c71-9dbd-a76427dbd05a None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:17.634841 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:17.642143 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 424/846] 10.222.0.90 () {58 vars in 1282 bytes} [Mon Aug 29 21:21:17 2022] GET /identity/v3/auth/tokens => generated 1161 bytes in 351 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 29 21:21:17.645175 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e640ff1-eb00-4c15-ae9a-13ad7d7c68fe None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:17.748047 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-369eec01-7318-4fa6-b6d3-d566661d1eb4 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:17.748321 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-369eec01-7318-4fa6-b6d3-d566661d1eb4 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:17.748593 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-369eec01-7318-4fa6-b6d3-d566661d1eb4 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:17.750185 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 425/847] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:21:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 29 21:21:17.755362 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e640ff1-eb00-4c15-ae9a-13ad7d7c68fe None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:17.757111 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09e9722e-e798-4f83-b1a1-6223f91f2c92 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:17.757398 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09e9722e-e798-4f83-b1a1-6223f91f2c92 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:17.757675 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09e9722e-e798-4f83-b1a1-6223f91f2c92 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:17.770443 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:17.780700 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-09e9722e-e798-4f83-b1a1-6223f91f2c92 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:21:17.782240 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-09e9722e-e798-4f83-b1a1-6223f91f2c92 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:21:17.802539 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc5ae085-db75-48f4-8620-2a9c24e8fe4f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:17.803661 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc5ae085-db75-48f4-8620-2a9c24e8fe4f None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:17.804427 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc5ae085-db75-48f4-8620-2a9c24e8fe4f None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:17.804894 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc5ae085-db75-48f4-8620-2a9c24e8fe4f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:17.819713 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc5ae085-db75-48f4-8620-2a9c24e8fe4f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:17.821846 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc5ae085-db75-48f4-8620-2a9c24e8fe4f None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:17.840715 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-dc5ae085-db75-48f4-8620-2a9c24e8fe4f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:21:17.840715 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:17.840715 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:17.840715 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:17.840715 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:17.840715 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:17.840715 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:17.840715 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:17.840715 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:17.840715 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:17.840715 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:17.840715 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:17.840715 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 21:21:17.840715 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 21:21:17.840715 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 21:21:17.840715 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 21:21:17.841898 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:17.841898 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:17.841898 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:17.841898 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:17.841898 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:17.841898 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:17.841898 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:17.841898 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:17.841898 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:17.841898 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:17.841898 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:17.841898 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:17.841898 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:17.841898 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:17.841898 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:17.841898 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:17.841898 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 21:21:17.842811 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 21:21:17.842811 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:21:17.842811 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:21:17.847486 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 423/848] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:21:17 2022] GET /identity/v3/domains/Default => generated 87 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:21:17.848633 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc5ae085-db75-48f4-8620-2a9c24e8fe4f None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:17.877102 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 426/849] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:21:17 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 29 21:21:17.966364 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc5ae085-db75-48f4-8620-2a9c24e8fe4f None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:18.002617 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31433378-0227-4bae-8552-a8f6f4325189 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:18.003521 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31433378-0227-4bae-8552-a8f6f4325189 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:18.003978 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31433378-0227-4bae-8552-a8f6f4325189 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:18.004427 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31433378-0227-4bae-8552-a8f6f4325189 None admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:18.005243 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31433378-0227-4bae-8552-a8f6f4325189 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:18.005745 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31433378-0227-4bae-8552-a8f6f4325189 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:18.007293 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31433378-0227-4bae-8552-a8f6f4325189 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:18.020934 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 424/850] 10.222.0.90 () {56 vars in 1109 bytes} [Mon Aug 29 21:21:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:18.026881 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09e9722e-e798-4f83-b1a1-6223f91f2c92 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:18.134595 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09e9722e-e798-4f83-b1a1-6223f91f2c92 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:18.159492 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31433378-0227-4bae-8552-a8f6f4325189 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:18.162936 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:18.176207 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bec3a24-b1c6-4faf-84c5-65e192a0b436 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:18.176847 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bec3a24-b1c6-4faf-84c5-65e192a0b436 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:18.177124 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bec3a24-b1c6-4faf-84c5-65e192a0b436 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:18.177391 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bec3a24-b1c6-4faf-84c5-65e192a0b436 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:18.187322 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bec3a24-b1c6-4faf-84c5-65e192a0b436 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:18.188745 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bec3a24-b1c6-4faf-84c5-65e192a0b436 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:18.195695 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:18.201727 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-6bec3a24-b1c6-4faf-84c5-65e192a0b436 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 21:21:18.201727 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:18.201727 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:18.201727 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:18.201727 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:18.201727 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:18.201727 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:18.201727 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:18.201727 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:18.201727 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:18.201727 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:18.201727 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:18.201727 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 29 21:21:18.201727 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 29 21:21:18.201727 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 29 21:21:18.206243 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 29 21:21:18.206243 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:18.206243 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:18.206243 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:18.206243 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:18.206243 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:18.206243 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:18.206243 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:18.206243 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:18.206243 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:18.206243 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:18.206243 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:18.206243 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:18.206243 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:18.206243 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:18.206243 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:18.206243 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:18.207186 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 29 21:21:18.207186 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 29 21:21:18.207186 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 29 21:21:18.207186 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 29 21:21:18.207186 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 29 21:21:18.207186 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 29 21:21:18.207186 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 21:21:18.207186 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:21:18.207186 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 427/851] 10.222.0.90 () {56 vars in 1111 bytes} [Mon Aug 29 21:21:18 2022] GET /identity/v3/projects/service => generated 88 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:21:18.208911 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bec3a24-b1c6-4faf-84c5-65e192a0b436 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:18.301076 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31433378-0227-4bae-8552-a8f6f4325189 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:18.319045 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bec3a24-b1c6-4faf-84c5-65e192a0b436 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:18.337475 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98fd44e0-23d2-4aea-8269-859590e0fb01 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:18.338428 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98fd44e0-23d2-4aea-8269-859590e0fb01 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:18.338946 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98fd44e0-23d2-4aea-8269-859590e0fb01 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:18.339073 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98fd44e0-23d2-4aea-8269-859590e0fb01 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:18.339564 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98fd44e0-23d2-4aea-8269-859590e0fb01 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:18.340940 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98fd44e0-23d2-4aea-8269-859590e0fb01 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:18.351087 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-98fd44e0-23d2-4aea-8269-859590e0fb01 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 29 21:21:18.351087 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:18.351087 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 29 21:21:18.351087 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 29 21:21:18.351087 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 29 21:21:18.351087 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 29 21:21:18.351087 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 29 21:21:18.351087 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 21:21:18.351087 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 29 21:21:18.351087 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:21:18.351087 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 29 21:21:18.351087 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:21:18.351087 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:18.351087 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:18.351087 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:18.351087 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:18.352266 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:18.352266 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:18.352266 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:18.352266 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:18.352266 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:18.352266 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:18.352266 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:18.352266 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 29 21:21:18.352266 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 29 21:21:18.352266 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 29 21:21:18.352266 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 29 21:21:18.352266 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:18.352266 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:18.352266 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:18.352266 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:18.352266 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:18.352266 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:18.353187 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:18.353187 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:18.353187 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:18.353187 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:18.353187 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:18.353187 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:18.353187 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:18.353187 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:18.353187 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:18.353187 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:18.353187 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 29 21:21:18.353187 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 21:21:18.353187 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 29 21:21:18.353187 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:21:18.353187 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 425/852] 10.222.0.90 () {56 vars in 1132 bytes} [Mon Aug 29 21:21:18 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 29 21:21:18.354016 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cf2a753-3cb1-448e-a65d-b377eaf502a7 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:18.354016 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cf2a753-3cb1-448e-a65d-b377eaf502a7 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:18.354367 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cf2a753-3cb1-448e-a65d-b377eaf502a7 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:18.355609 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 426/853] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:21:18 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:21:18.361003 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cf2a753-3cb1-448e-a65d-b377eaf502a7 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:18.366674 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dd5e975-3023-4c41-82d4-53e5ffcc9e81 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:18.367292 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dd5e975-3023-4c41-82d4-53e5ffcc9e81 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:18.367572 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dd5e975-3023-4c41-82d4-53e5ffcc9e81 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:18.367846 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dd5e975-3023-4c41-82d4-53e5ffcc9e81 None admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:18.368539 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dd5e975-3023-4c41-82d4-53e5ffcc9e81 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:18.368837 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dd5e975-3023-4c41-82d4-53e5ffcc9e81 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:18.370331 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dd5e975-3023-4c41-82d4-53e5ffcc9e81 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:18.383352 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 428/854] 10.222.0.90 () {56 vars in 1148 bytes} [Mon Aug 29 21:21:18 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:18.389148 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e0f9652-b415-4758-a14b-d5d6a8c6c798 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:18.389440 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e0f9652-b415-4758-a14b-d5d6a8c6c798 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:18.389712 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e0f9652-b415-4758-a14b-d5d6a8c6c798 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:18.408517 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-2e0f9652-b415-4758-a14b-d5d6a8c6c798 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:21:18.409713 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-2e0f9652-b415-4758-a14b-d5d6a8c6c798 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:21:18.454482 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:18.460044 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:18.477573 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cf2a753-3cb1-448e-a65d-b377eaf502a7 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:18.485956 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 429/855] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:21:18 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 21:21:18.487107 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e0f9652-b415-4758-a14b-d5d6a8c6c798 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:18.528741 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29c22c9a-2f07-495f-967b-dc39ed2d2d4f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:18.529941 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29c22c9a-2f07-495f-967b-dc39ed2d2d4f None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:18.530411 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29c22c9a-2f07-495f-967b-dc39ed2d2d4f None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:18.530861 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29c22c9a-2f07-495f-967b-dc39ed2d2d4f None admin] PATH_INFO: `/v3/services` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:18.531706 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29c22c9a-2f07-495f-967b-dc39ed2d2d4f None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:18.532224 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29c22c9a-2f07-495f-967b-dc39ed2d2d4f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:18.534440 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29c22c9a-2f07-495f-967b-dc39ed2d2d4f None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:18.542068 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 427/856] 10.222.0.90 () {56 vars in 1126 bytes} [Mon Aug 29 21:21:18 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:18.547602 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29c22c9a-2f07-495f-967b-dc39ed2d2d4f None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:18.617551 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e0f9652-b415-4758-a14b-d5d6a8c6c798 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:18.667945 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c451509b-bf2e-4b50-a8f3-6b2afa9e4700 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:18.669115 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c451509b-bf2e-4b50-a8f3-6b2afa9e4700 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:18.669923 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c451509b-bf2e-4b50-a8f3-6b2afa9e4700 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:18.670380 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c451509b-bf2e-4b50-a8f3-6b2afa9e4700 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:18.671926 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c451509b-bf2e-4b50-a8f3-6b2afa9e4700 None admin] RBAC: Adding query filter params (role.id=a6e562cdc96c4be691d638990f8403fd, scope.project.id=ec03e112e3d547fbab6faf7b485c3fca, user.id=139bbbde6d884b42b5440727926e2c58) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:18.672438 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c451509b-bf2e-4b50-a8f3-6b2afa9e4700 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:18.675618 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c451509b-bf2e-4b50-a8f3-6b2afa9e4700 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:18.677433 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29c22c9a-2f07-495f-967b-dc39ed2d2d4f None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:18.683714 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 430/857] 10.222.0.90 () {56 vars in 1374 bytes} [Mon Aug 29 21:21:18 2022] GET /identity/v3/role_assignments?user.id=139bbbde6d884b42b5440727926e2c58&scope.project.id=ec03e112e3d547fbab6faf7b485c3fca&role.id=a6e562cdc96c4be691d638990f8403fd => generated 608 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:18.728443 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-361fedab-ae95-44c0-b660-4e5c103ad7fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:18.729066 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-361fedab-ae95-44c0-b660-4e5c103ad7fc None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:18.729348 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-361fedab-ae95-44c0-b660-4e5c103ad7fc None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:18.729682 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-361fedab-ae95-44c0-b660-4e5c103ad7fc None admin] PATH_INFO: `/v3/services` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:18.730371 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-361fedab-ae95-44c0-b660-4e5c103ad7fc None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:18.730708 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-361fedab-ae95-44c0-b660-4e5c103ad7fc None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:18.732098 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-361fedab-ae95-44c0-b660-4e5c103ad7fc None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:18.742074 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 428/858] 10.222.0.90 () {56 vars in 1126 bytes} [Mon Aug 29 21:21:18 2022] GET /identity/v3/services?type=compute_legacy => generated 131 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:21:18.768886 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c451509b-bf2e-4b50-a8f3-6b2afa9e4700 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:18.823167 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:18.908746 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c451509b-bf2e-4b50-a8f3-6b2afa9e4700 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:18.945741 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91b0d94a-6995-46ca-b4a4-4d9421083c27 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:18.946417 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91b0d94a-6995-46ca-b4a4-4d9421083c27 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:18.946697 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91b0d94a-6995-46ca-b4a4-4d9421083c27 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:18.946973 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91b0d94a-6995-46ca-b4a4-4d9421083c27 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:18.960038 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91b0d94a-6995-46ca-b4a4-4d9421083c27 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 21:21:18.965277 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-91b0d94a-6995-46ca-b4a4-4d9421083c27 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 29 21:21:18.965277 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:18.965277 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:18.965277 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:18.965277 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:18.965277 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:18.965277 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:18.965277 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:18.965277 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:18.965277 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:18.965277 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:18.965277 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:18.965277 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 21:21:18.965277 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 21:21:18.965277 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 21:21:18.965277 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 21:21:18.966277 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 21:21:18.966277 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 21:21:18.966277 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 21:21:18.966277 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 21:21:18.966277 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:18.966277 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:18.966277 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:18.966277 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:18.966277 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:18.966277 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:18.966277 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:18.966277 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:18.966277 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:18.966277 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:18.966277 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:18.966277 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:18.967776 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:18.967776 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:18.967776 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:18.967776 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:18.967776 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 21:21:18.967776 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 21:21:18.967776 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 21:21:18.967776 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 21:21:18.967776 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 21:21:18.967776 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 21:21:18.967776 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 29 21:21:18.967776 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:21:18.967776 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 431/859] 10.222.0.90 () {56 vars in 1102 bytes} [Mon Aug 29 21:21:18 2022] GET /identity/v3/roles/service => generated 85 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:21:18.973770 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-361fedab-ae95-44c0-b660-4e5c103ad7fc None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:18.994050 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:19.088115 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-361fedab-ae95-44c0-b660-4e5c103ad7fc None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:19.123220 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c734410e-b906-4550-a556-349fefc18d73 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:19.123837 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c734410e-b906-4550-a556-349fefc18d73 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:19.124118 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c734410e-b906-4550-a556-349fefc18d73 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:19.124394 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c734410e-b906-4550-a556-349fefc18d73 None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:19.125113 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c734410e-b906-4550-a556-349fefc18d73 None admin] RBAC: Adding query filter params (name=service) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:19.125421 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c734410e-b906-4550-a556-349fefc18d73 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:19.127412 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c734410e-b906-4550-a556-349fefc18d73 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:19.136218 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 429/860] 10.222.0.90 () {56 vars in 1103 bytes} [Mon Aug 29 21:21:18 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:19.141732 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91b0d94a-6995-46ca-b4a4-4d9421083c27 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:19.247015 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91b0d94a-6995-46ca-b4a4-4d9421083c27 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:19.248872 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:19.287919 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1d4e39d-0fe8-4b86-a725-e5838acf3547 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:19.288564 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1d4e39d-0fe8-4b86-a725-e5838acf3547 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:19.288848 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1d4e39d-0fe8-4b86-a725-e5838acf3547 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:19.289125 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1d4e39d-0fe8-4b86-a725-e5838acf3547 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:19.301636 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1d4e39d-0fe8-4b86-a725-e5838acf3547 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:19.303204 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1d4e39d-0fe8-4b86-a725-e5838acf3547 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:19.318588 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-f1d4e39d-0fe8-4b86-a725-e5838acf3547 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:21:19.318588 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:19.318588 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:19.318588 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:19.318588 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:19.318588 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:19.318588 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:19.318588 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:19.318588 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:19.318588 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:19.318588 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:19.318588 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:19.318588 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 21:21:19.318588 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 21:21:19.318588 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 21:21:19.318588 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 21:21:19.320004 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:19.320004 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:19.320004 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:19.320004 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:19.320004 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:19.320004 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:19.320004 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:19.320004 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:19.320004 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:19.320004 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:19.320004 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:19.320004 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:19.320004 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:19.320004 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:19.320004 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:19.320004 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:19.320004 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 21:21:19.320918 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 21:21:19.320918 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:21:19.320918 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:21:19.320918 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 432/861] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:21:19 2022] GET /identity/v3/domains/Default => generated 87 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:21:19.327304 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c734410e-b906-4550-a556-349fefc18d73 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:19.407600 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e14dcfb5-5d3e-419f-b3d6-f0ce8389a29b None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:19.407952 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e14dcfb5-5d3e-419f-b3d6-f0ce8389a29b None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:19.408280 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e14dcfb5-5d3e-419f-b3d6-f0ce8389a29b None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:19.409881 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 433/862] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:21:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 29 21:21:19.417191 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5251640-45e3-4805-85a2-abfb7cac3591 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:19.417768 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5251640-45e3-4805-85a2-abfb7cac3591 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:19.418274 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5251640-45e3-4805-85a2-abfb7cac3591 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:19.424935 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:19.438465 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-d5251640-45e3-4805-85a2-abfb7cac3591 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:21:19.441847 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-d5251640-45e3-4805-85a2-abfb7cac3591 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:21:19.455376 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c734410e-b906-4550-a556-349fefc18d73 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:19.497825 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96f3db58-fd62-4e35-a33d-33cc78b7538b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:19.498424 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96f3db58-fd62-4e35-a33d-33cc78b7538b None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:19.498713 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96f3db58-fd62-4e35-a33d-33cc78b7538b None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:19.498985 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96f3db58-fd62-4e35-a33d-33cc78b7538b None admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:19.499790 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96f3db58-fd62-4e35-a33d-33cc78b7538b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:19.500100 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96f3db58-fd62-4e35-a33d-33cc78b7538b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:19.502379 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96f3db58-fd62-4e35-a33d-33cc78b7538b None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:19.517883 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 430/863] 10.222.0.90 () {56 vars in 1109 bytes} [Mon Aug 29 21:21:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:19.525919 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 434/864] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:21:19 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 29 21:21:19.526061 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96f3db58-fd62-4e35-a33d-33cc78b7538b None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:19.627972 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96f3db58-fd62-4e35-a33d-33cc78b7538b None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:19.663200 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ec872ab-00c6-4b20-ab57-9ef7e2033595 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:19.663837 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec872ab-00c6-4b20-ab57-9ef7e2033595 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:19.664161 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec872ab-00c6-4b20-ab57-9ef7e2033595 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:19.664442 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec872ab-00c6-4b20-ab57-9ef7e2033595 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:19.666381 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:19.673197 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ec872ab-00c6-4b20-ab57-9ef7e2033595 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:19.675649 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ec872ab-00c6-4b20-ab57-9ef7e2033595 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:19.687387 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-8ec872ab-00c6-4b20-ab57-9ef7e2033595 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 29 21:21:19.687387 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:19.687387 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:19.687387 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:19.687387 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:19.687387 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:19.687387 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:19.687387 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:19.687387 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:19.687387 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:19.687387 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:19.687387 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:19.687387 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 29 21:21:19.687387 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 29 21:21:19.687387 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 29 21:21:19.687387 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 29 21:21:19.688330 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:19.688330 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:19.688330 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 29 21:21:19.688330 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 21:21:19.688330 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 29 21:21:19.688330 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 21:21:19.688330 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:19.688330 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:19.688330 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:19.688330 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:19.688330 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:19.688330 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:19.688330 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:19.688330 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:19.688330 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:19.688330 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:19.688330 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:19.688330 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:19.690314 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:19.690314 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:19.690314 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 29 21:21:19.690314 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 29 21:21:19.690314 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 29 21:21:19.690314 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 29 21:21:19.690314 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 29 21:21:19.690314 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 29 21:21:19.690314 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 29 21:21:19.690314 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:21:19.690314 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 431/865] 10.222.0.90 () {56 vars in 1117 bytes} [Mon Aug 29 21:21:19 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 29 21:21:19.698906 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5251640-45e3-4805-85a2-abfb7cac3591 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:19.786705 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:19.806271 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5251640-45e3-4805-85a2-abfb7cac3591 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:19.837464 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ec872ab-00c6-4b20-ab57-9ef7e2033595 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:19.845840 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27c0da9d-f758-4321-b76e-b81035dfe00c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:19.845840 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27c0da9d-f758-4321-b76e-b81035dfe00c None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:19.845840 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27c0da9d-f758-4321-b76e-b81035dfe00c None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:19.845840 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27c0da9d-f758-4321-b76e-b81035dfe00c None admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:19.846520 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27c0da9d-f758-4321-b76e-b81035dfe00c None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:19.846520 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27c0da9d-f758-4321-b76e-b81035dfe00c None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:19.846520 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27c0da9d-f758-4321-b76e-b81035dfe00c None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:19.867176 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 435/866] 10.222.0.90 () {56 vars in 1149 bytes} [Mon Aug 29 21:21:19 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:19.875450 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27c0da9d-f758-4321-b76e-b81035dfe00c None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:19.971292 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ec872ab-00c6-4b20-ab57-9ef7e2033595 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:20.004904 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27c0da9d-f758-4321-b76e-b81035dfe00c None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:20.008686 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e8e1b85-c71d-4351-9ba7-70fbeb21aa0c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:20.009307 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e8e1b85-c71d-4351-9ba7-70fbeb21aa0c None admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:20.009583 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e8e1b85-c71d-4351-9ba7-70fbeb21aa0c None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:20.009893 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e8e1b85-c71d-4351-9ba7-70fbeb21aa0c None admin] PATH_INFO: `/v3/services` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:20.010560 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e8e1b85-c71d-4351-9ba7-70fbeb21aa0c None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:20.011917 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e8e1b85-c71d-4351-9ba7-70fbeb21aa0c None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:20.031035 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 432/867] 10.222.0.90 () {60 vars in 1141 bytes} [Mon Aug 29 21:21:19 2022] POST /identity/v3/services => generated 251 bytes in 194 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:21:20.054298 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7b004df-f3ae-401f-920e-c1e98cd3911e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:20.054854 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7b004df-f3ae-401f-920e-c1e98cd3911e None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:20.055185 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7b004df-f3ae-401f-920e-c1e98cd3911e None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:20.055461 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7b004df-f3ae-401f-920e-c1e98cd3911e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:20.065465 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:20.137641 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7b004df-f3ae-401f-920e-c1e98cd3911e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:20.139147 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7b004df-f3ae-401f-920e-c1e98cd3911e None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:20.246946 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 436/868] 10.222.0.90 () {58 vars in 1282 bytes} [Mon Aug 29 21:21:19 2022] GET /identity/v3/auth/tokens => generated 1262 bytes in 372 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Aug 29 21:21:20.252721 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e8e1b85-c71d-4351-9ba7-70fbeb21aa0c None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:20.311851 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:20.360443 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e8e1b85-c71d-4351-9ba7-70fbeb21aa0c None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:20.396800 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f782ff95-b367-46ab-9d44-687b1a954088 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:20.397439 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f782ff95-b367-46ab-9d44-687b1a954088 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:20.397715 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f782ff95-b367-46ab-9d44-687b1a954088 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:20.398010 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f782ff95-b367-46ab-9d44-687b1a954088 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:20.409906 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f782ff95-b367-46ab-9d44-687b1a954088 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:20.411362 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f782ff95-b367-46ab-9d44-687b1a954088 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:20.426896 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-f782ff95-b367-46ab-9d44-687b1a954088 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:21:20.426896 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:20.426896 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:20.426896 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:20.426896 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:20.426896 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:20.426896 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:20.426896 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:20.426896 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:20.426896 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:20.426896 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:20.426896 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:20.426896 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 21:21:20.426896 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 21:21:20.426896 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 21:21:20.426896 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 21:21:20.429370 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:20.429370 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:20.429370 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:20.429370 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:20.429370 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:20.429370 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:20.429370 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:20.429370 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:20.429370 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:20.429370 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:20.429370 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:20.429370 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:20.429370 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:20.429370 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:20.429370 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:20.429370 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:20.429370 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 21:21:20.431596 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 21:21:20.431596 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:21:20.431596 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:21:20.431596 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 433/869] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:21:20 2022] GET /identity/v3/domains/Default => generated 87 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 29 21:21:20.435094 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7b004df-f3ae-401f-920e-c1e98cd3911e None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:20.482488 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:20.543813 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7b004df-f3ae-401f-920e-c1e98cd3911e None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:20.586609 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17a71b60-6ad3-437a-a4ea-4ef762b7dfa2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:20.589813 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17a71b60-6ad3-437a-a4ea-4ef762b7dfa2 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:20.589813 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17a71b60-6ad3-437a-a4ea-4ef762b7dfa2 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:20.589813 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17a71b60-6ad3-437a-a4ea-4ef762b7dfa2 None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:20.589813 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17a71b60-6ad3-437a-a4ea-4ef762b7dfa2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:20.589813 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17a71b60-6ad3-437a-a4ea-4ef762b7dfa2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:20.590605 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17a71b60-6ad3-437a-a4ea-4ef762b7dfa2 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:20.603501 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 437/870] 10.222.0.90 () {56 vars in 1109 bytes} [Mon Aug 29 21:21:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:20.609137 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f782ff95-b367-46ab-9d44-687b1a954088 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:20.713935 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f782ff95-b367-46ab-9d44-687b1a954088 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:20.722024 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:20.750012 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e55c8ba-821b-4115-b39d-e32b6d012afe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:20.750661 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e55c8ba-821b-4115-b39d-e32b6d012afe None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:20.750929 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e55c8ba-821b-4115-b39d-e32b6d012afe None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:20.751170 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e55c8ba-821b-4115-b39d-e32b6d012afe None admin] PATH_INFO: `/v3/projects/service` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:20.759605 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e55c8ba-821b-4115-b39d-e32b6d012afe None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:20.761062 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e55c8ba-821b-4115-b39d-e32b6d012afe None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:20.763833 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa093948-d228-47dd-84ec-b6ca5c0ae7f7 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:20.764122 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa093948-d228-47dd-84ec-b6ca5c0ae7f7 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:20.764400 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa093948-d228-47dd-84ec-b6ca5c0ae7f7 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:20.766019 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 438/871] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:21:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 29 21:21:20.775004 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19444cd6-881b-4c57-b2e2-96085531161f None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:20.775004 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19444cd6-881b-4c57-b2e2-96085531161f None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:20.775004 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19444cd6-881b-4c57-b2e2-96085531161f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:20.775510 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-8e55c8ba-821b-4115-b39d-e32b6d012afe None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 21:21:20.775510 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:20.775510 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:20.775510 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:20.775510 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:20.775510 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:20.775510 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:20.775510 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:20.775510 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:20.775510 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:20.775510 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:20.775510 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:20.775510 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 29 21:21:20.775510 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 29 21:21:20.775510 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 29 21:21:20.777675 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 29 21:21:20.777675 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:20.777675 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:20.777675 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:20.777675 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:20.777675 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:20.777675 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:20.777675 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:20.777675 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:20.777675 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:20.777675 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:20.777675 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:20.777675 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:20.777675 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:20.777675 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:20.777675 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:20.777675 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:20.779951 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 29 21:21:20.779951 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 29 21:21:20.779951 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 29 21:21:20.779951 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 29 21:21:20.779951 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 29 21:21:20.779951 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 29 21:21:20.779951 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 21:21:20.779951 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:21:20.779951 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 434/872] 10.222.0.90 () {56 vars in 1111 bytes} [Mon Aug 29 21:21:20 2022] GET /identity/v3/projects/service => generated 88 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:21:20.781677 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e55c8ba-821b-4115-b39d-e32b6d012afe None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:20.798157 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-19444cd6-881b-4c57-b2e2-96085531161f None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:21:20.799968 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-19444cd6-881b-4c57-b2e2-96085531161f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:21:20.886787 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 439/873] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:21:20 2022] POST /identity/v3/auth/tokens => generated 1262 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 21:21:20.902389 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-527c9353-3245-4e84-af24-c683d48aef9b None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:20.902661 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-527c9353-3245-4e84-af24-c683d48aef9b None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:20.902920 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-527c9353-3245-4e84-af24-c683d48aef9b None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:20.903569 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e55c8ba-821b-4115-b39d-e32b6d012afe None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:20.904459 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 440/874] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:21:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:21:20.910362 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60a97848-632d-47a9-a259-5a6a9804ae82 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:20.910698 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60a97848-632d-47a9-a259-5a6a9804ae82 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:20.910976 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60a97848-632d-47a9-a259-5a6a9804ae82 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:20.931722 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-60a97848-632d-47a9-a259-5a6a9804ae82 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:21:20.932910 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-60a97848-632d-47a9-a259-5a6a9804ae82 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:21:20.942073 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c94e155-931a-435e-ad7c-e307cfc2e081 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:20.942697 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c94e155-931a-435e-ad7c-e307cfc2e081 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:20.942979 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c94e155-931a-435e-ad7c-e307cfc2e081 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:20.943355 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c94e155-931a-435e-ad7c-e307cfc2e081 None admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:20.944065 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c94e155-931a-435e-ad7c-e307cfc2e081 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:20.944369 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c94e155-931a-435e-ad7c-e307cfc2e081 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:20.945944 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c94e155-931a-435e-ad7c-e307cfc2e081 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:20.957987 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:20.961340 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 435/875] 10.222.0.90 () {56 vars in 1148 bytes} [Mon Aug 29 21:21:20 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:21:20.967978 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c94e155-931a-435e-ad7c-e307cfc2e081 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:21.007243 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:21.029962 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 441/876] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:21:20 2022] POST /identity/v3/auth/tokens => generated 1262 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 29 21:21:21.091626 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c94e155-931a-435e-ad7c-e307cfc2e081 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:21.129838 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-463d57f1-3425-42e1-99ac-e8c97dc21699 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:21.130688 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-463d57f1-3425-42e1-99ac-e8c97dc21699 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:21.131110 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-463d57f1-3425-42e1-99ac-e8c97dc21699 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:21.131523 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-463d57f1-3425-42e1-99ac-e8c97dc21699 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:21.132387 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-463d57f1-3425-42e1-99ac-e8c97dc21699 None admin] RBAC: Adding query filter params (role.id=a94bdb4c5796428baf63644c062dfcb2, scope.project.id=ec03e112e3d547fbab6faf7b485c3fca, user.id=0c0b0da87b124de1be00faec4bbdb404) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:21.132853 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-463d57f1-3425-42e1-99ac-e8c97dc21699 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:21.134430 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-463d57f1-3425-42e1-99ac-e8c97dc21699 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:21.140899 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 436/877] 10.222.0.90 () {56 vars in 1374 bytes} [Mon Aug 29 21:21:20 2022] GET /identity/v3/role_assignments?user.id=0c0b0da87b124de1be00faec4bbdb404&scope.project.id=ec03e112e3d547fbab6faf7b485c3fca&role.id=a94bdb4c5796428baf63644c062dfcb2 => generated 608 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:21.175438 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60a97848-632d-47a9-a259-5a6a9804ae82 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:21.301367 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60a97848-632d-47a9-a259-5a6a9804ae82 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:21.313551 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-463d57f1-3425-42e1-99ac-e8c97dc21699 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:21.342464 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fb77315-be5e-4c55-9978-7b62279cb7f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:21.343340 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fb77315-be5e-4c55-9978-7b62279cb7f9 None admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:21.343624 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fb77315-be5e-4c55-9978-7b62279cb7f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:21.343896 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fb77315-be5e-4c55-9978-7b62279cb7f9 None admin] PATH_INFO: `/v3/services` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:21.344568 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fb77315-be5e-4c55-9978-7b62279cb7f9 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:21.346334 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fb77315-be5e-4c55-9978-7b62279cb7f9 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:21.369928 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 442/878] 10.222.0.90 () {60 vars in 1141 bytes} [Mon Aug 29 21:21:21 2022] POST /identity/v3/services => generated 269 bytes in 189 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:21:21.433681 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:21.449987 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-463d57f1-3425-42e1-99ac-e8c97dc21699 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:21.497047 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-303a895c-e647-44ab-a200-0193da3a5717 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:21.497687 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-303a895c-e647-44ab-a200-0193da3a5717 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:21.497991 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-303a895c-e647-44ab-a200-0193da3a5717 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:21.498304 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-303a895c-e647-44ab-a200-0193da3a5717 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:21.498905 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-303a895c-e647-44ab-a200-0193da3a5717 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:21.500284 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-303a895c-e647-44ab-a200-0193da3a5717 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:21.511008 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-303a895c-e647-44ab-a200-0193da3a5717 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 29 21:21:21.511008 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:21.511008 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 29 21:21:21.511008 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 29 21:21:21.511008 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 29 21:21:21.511008 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 29 21:21:21.511008 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 29 21:21:21.511008 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 21:21:21.511008 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 29 21:21:21.511008 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:21:21.511008 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 29 21:21:21.511008 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:21:21.511008 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:21.511008 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:21.511008 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:21.511008 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:21.511008 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:21.512084 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:21.512084 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:21.512084 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:21.512084 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:21.512084 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:21.512084 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:21.512084 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 29 21:21:21.512084 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 29 21:21:21.512084 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 29 21:21:21.512084 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 29 21:21:21.512084 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:21.512084 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:21.512084 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:21.512084 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:21.512084 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:21.512084 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:21.512935 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:21.512935 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:21.512935 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:21.512935 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:21.512935 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:21.512935 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:21.512935 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:21.512935 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:21.512935 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:21.512935 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:21.512935 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 29 21:21:21.512935 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 21:21:21.512935 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 29 21:21:21.512935 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:21:21.512935 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 437/879] 10.222.0.90 () {56 vars in 1126 bytes} [Mon Aug 29 21:21:21 2022] GET /identity/v3/services/object-store => generated 93 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:21:21.519185 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fb77315-be5e-4c55-9978-7b62279cb7f9 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:21.540973 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:21.634481 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fb77315-be5e-4c55-9978-7b62279cb7f9 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:21.669320 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f4d8eea-2db5-4eb2-a94e-de0d27d7f78c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:21.670050 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f4d8eea-2db5-4eb2-a94e-de0d27d7f78c None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:21.670332 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f4d8eea-2db5-4eb2-a94e-de0d27d7f78c None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:21.670602 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f4d8eea-2db5-4eb2-a94e-de0d27d7f78c None admin] PATH_INFO: `/v3/services` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:21.671235 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f4d8eea-2db5-4eb2-a94e-de0d27d7f78c None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:21.671537 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f4d8eea-2db5-4eb2-a94e-de0d27d7f78c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:21.672908 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f4d8eea-2db5-4eb2-a94e-de0d27d7f78c None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:21.679464 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 443/880] 10.222.0.90 () {56 vars in 1122 bytes} [Mon Aug 29 21:21:21 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:21.685127 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-303a895c-e647-44ab-a200-0193da3a5717 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:21.791118 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-303a895c-e647-44ab-a200-0193da3a5717 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:21.826268 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ed70fe0-2248-4745-a4f7-0959c6149e44 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:21.826882 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ed70fe0-2248-4745-a4f7-0959c6149e44 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:21.827158 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ed70fe0-2248-4745-a4f7-0959c6149e44 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:21.827431 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ed70fe0-2248-4745-a4f7-0959c6149e44 None admin] PATH_INFO: `/v3/services` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:21.828062 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ed70fe0-2248-4745-a4f7-0959c6149e44 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:21.828363 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ed70fe0-2248-4745-a4f7-0959c6149e44 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:21.830303 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ed70fe0-2248-4745-a4f7-0959c6149e44 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:21.835622 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 438/881] 10.222.0.90 () {56 vars in 1122 bytes} [Mon Aug 29 21:21:21 2022] GET /identity/v3/services?type=object-store => generated 129 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:22.076519 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f21ade7-97d4-4ed4-af10-5a1889560866 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:22.076913 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f21ade7-97d4-4ed4-af10-5a1889560866 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:22.077095 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f21ade7-97d4-4ed4-af10-5a1889560866 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:22.078879 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 444/882] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:21:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:21:22.084932 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f346633-32c2-41b2-9109-c7b75c8edeed None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:22.085479 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f346633-32c2-41b2-9109-c7b75c8edeed None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:22.086020 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f346633-32c2-41b2-9109-c7b75c8edeed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:22.104755 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-6f346633-32c2-41b2-9109-c7b75c8edeed None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:21:22.106306 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-6f346633-32c2-41b2-9109-c7b75c8edeed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:21:22.116822 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:22.177509 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 439/883] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:21:22 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 21:21:22.446989 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f21ade7-97d4-4ed4-af10-5a1889560866 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:22.533928 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:22.558117 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f21ade7-97d4-4ed4-af10-5a1889560866 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:22.596915 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb102fac-eea3-4f43-a32b-cc2c7f9eaa33 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:22.597560 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb102fac-eea3-4f43-a32b-cc2c7f9eaa33 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:22.597870 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb102fac-eea3-4f43-a32b-cc2c7f9eaa33 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:22.598152 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb102fac-eea3-4f43-a32b-cc2c7f9eaa33 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:22.598746 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb102fac-eea3-4f43-a32b-cc2c7f9eaa33 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:22.600112 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb102fac-eea3-4f43-a32b-cc2c7f9eaa33 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:22.611642 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-fb102fac-eea3-4f43-a32b-cc2c7f9eaa33 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 29 21:21:22.611642 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:22.611642 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 29 21:21:22.611642 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 29 21:21:22.611642 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 29 21:21:22.611642 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 29 21:21:22.611642 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 29 21:21:22.611642 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 21:21:22.611642 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 29 21:21:22.611642 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:21:22.611642 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 29 21:21:22.611642 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:21:22.611642 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:22.611642 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:22.611642 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:22.611642 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:22.611642 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:22.612637 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:22.612637 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:22.612637 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:22.612637 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:22.612637 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:22.612637 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:22.612637 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 29 21:21:22.612637 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 29 21:21:22.612637 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 29 21:21:22.612637 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 29 21:21:22.612637 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:22.612637 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:22.612637 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:22.612637 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:22.612637 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:22.612637 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:22.613502 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:22.613502 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:22.613502 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:22.613502 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:22.613502 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:22.613502 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:22.613502 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:22.613502 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:22.613502 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:22.613502 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:22.613502 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 29 21:21:22.613502 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 21:21:22.613502 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 29 21:21:22.613502 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:21:22.613502 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 445/884] 10.222.0.90 () {56 vars in 1114 bytes} [Mon Aug 29 21:21:22 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:21:22.618718 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f346633-32c2-41b2-9109-c7b75c8edeed None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:22.719440 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f346633-32c2-41b2-9109-c7b75c8edeed None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:22.755861 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-362c5942-b307-4e27-bd56-f98cb96b2f03 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:22.756980 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-362c5942-b307-4e27-bd56-f98cb96b2f03 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:22.757471 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-362c5942-b307-4e27-bd56-f98cb96b2f03 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:22.757990 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-362c5942-b307-4e27-bd56-f98cb96b2f03 None admin] PATH_INFO: `/v3/services` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:22.758820 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-362c5942-b307-4e27-bd56-f98cb96b2f03 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:22.759316 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-362c5942-b307-4e27-bd56-f98cb96b2f03 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:22.760870 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-362c5942-b307-4e27-bd56-f98cb96b2f03 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:22.766324 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 440/885] 10.222.0.90 () {56 vars in 1114 bytes} [Mon Aug 29 21:21:22 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:22.773912 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb102fac-eea3-4f43-a32b-cc2c7f9eaa33 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:22.877341 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb102fac-eea3-4f43-a32b-cc2c7f9eaa33 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:22.912721 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e99c9e73-4281-436f-ae32-e10d8ea19d69 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:22.913334 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e99c9e73-4281-436f-ae32-e10d8ea19d69 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:22.913610 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e99c9e73-4281-436f-ae32-e10d8ea19d69 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:22.913911 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e99c9e73-4281-436f-ae32-e10d8ea19d69 None admin] PATH_INFO: `/v3/services` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:22.914686 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e99c9e73-4281-436f-ae32-e10d8ea19d69 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:22.914982 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e99c9e73-4281-436f-ae32-e10d8ea19d69 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:22.916340 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e99c9e73-4281-436f-ae32-e10d8ea19d69 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:22.921758 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 446/886] 10.222.0.90 () {56 vars in 1114 bytes} [Mon Aug 29 21:21:22 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:22.927996 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-362c5942-b307-4e27-bd56-f98cb96b2f03 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:22.954180 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:23.036625 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-362c5942-b307-4e27-bd56-f98cb96b2f03 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:23.072724 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a346e39c-ca9f-4106-b9dd-644da154a40a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:23.073333 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a346e39c-ca9f-4106-b9dd-644da154a40a None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:23.073612 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a346e39c-ca9f-4106-b9dd-644da154a40a None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:23.073908 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a346e39c-ca9f-4106-b9dd-644da154a40a None admin] PATH_INFO: `/v3/endpoints` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:23.074615 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a346e39c-ca9f-4106-b9dd-644da154a40a None admin] RBAC: Adding query filter params (interface=public, service_id=0ab1be01e887484682fff9b1de356916, region_id=RegionOne) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:23.074919 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a346e39c-ca9f-4106-b9dd-644da154a40a None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:23.076347 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a346e39c-ca9f-4106-b9dd-644da154a40a None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:23.082594 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 441/887] 10.222.0.90 () {56 vars in 1251 bytes} [Mon Aug 29 21:21:22 2022] GET /identity/v3/endpoints?service_id=0ab1be01e887484682fff9b1de356916&interface=public®ion_id=RegionOne => generated 194 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:21:23.087728 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e99c9e73-4281-436f-ae32-e10d8ea19d69 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:23.100976 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:23.194768 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e99c9e73-4281-436f-ae32-e10d8ea19d69 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:23.230474 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8f09452-16e1-465e-b8b3-b654a666db56 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:23.231416 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8f09452-16e1-465e-b8b3-b654a666db56 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:23.231911 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8f09452-16e1-465e-b8b3-b654a666db56 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:23.232389 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8f09452-16e1-465e-b8b3-b654a666db56 None admin] PATH_INFO: `/v3/services` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:23.233165 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8f09452-16e1-465e-b8b3-b654a666db56 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:23.234725 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8f09452-16e1-465e-b8b3-b654a666db56 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:23.241670 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 447/888] 10.222.0.90 () {56 vars in 1087 bytes} [Mon Aug 29 21:21:23 2022] GET /identity/v3/services => generated 1268 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 29 21:21:23.344636 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df27c6f0-d43d-4392-b7c1-3183e5c9f97e None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:23.345488 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df27c6f0-d43d-4392-b7c1-3183e5c9f97e None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:23.346324 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df27c6f0-d43d-4392-b7c1-3183e5c9f97e None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:23.348228 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 442/889] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:21:23 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:21:23.354439 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52d9d099-1fe4-4a5a-aa50-1b87f046902f None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:23.355080 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52d9d099-1fe4-4a5a-aa50-1b87f046902f None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:23.355549 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52d9d099-1fe4-4a5a-aa50-1b87f046902f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:23.375898 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-52d9d099-1fe4-4a5a-aa50-1b87f046902f None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:21:23.378171 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-52d9d099-1fe4-4a5a-aa50-1b87f046902f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:21:23.456301 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 448/890] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:21:23 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 21:21:23.466140 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1eed9f2-9f47-497e-a5de-1c82ff307eab None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:23.466700 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1eed9f2-9f47-497e-a5de-1c82ff307eab None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:23.467171 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1eed9f2-9f47-497e-a5de-1c82ff307eab None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:23.468918 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 443/891] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:21:23 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:21:23.476202 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9344cad-31d1-4800-9b33-3d357aea4599 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:23.476849 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9344cad-31d1-4800-9b33-3d357aea4599 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:23.477324 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9344cad-31d1-4800-9b33-3d357aea4599 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:23.489809 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:23.498165 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-a9344cad-31d1-4800-9b33-3d357aea4599 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:21:23.499371 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-a9344cad-31d1-4800-9b33-3d357aea4599 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:21:23.570554 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 449/892] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:21:23 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 29 21:21:23.731237 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1eed9f2-9f47-497e-a5de-1c82ff307eab None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:23.809777 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:23.840780 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1eed9f2-9f47-497e-a5de-1c82ff307eab None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:23.866915 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9344cad-31d1-4800-9b33-3d357aea4599 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:23.890664 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-567c7cde-9e30-44df-8830-ed142ff55456 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:23.891372 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-567c7cde-9e30-44df-8830-ed142ff55456 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:23.891646 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-567c7cde-9e30-44df-8830-ed142ff55456 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:23.892006 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-567c7cde-9e30-44df-8830-ed142ff55456 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:23.906170 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-567c7cde-9e30-44df-8830-ed142ff55456 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 21:21:23.912832 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-567c7cde-9e30-44df-8830-ed142ff55456 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 29 21:21:23.912832 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:23.912832 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:23.912832 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:23.912832 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:23.912832 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:23.912832 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:23.912832 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:23.912832 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:23.912832 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:23.912832 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:23.912832 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:23.912832 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 21:21:23.912832 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 21:21:23.912832 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 21:21:23.912832 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 21:21:23.913773 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 21:21:23.913773 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 21:21:23.913773 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 21:21:23.913773 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 21:21:23.913773 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:23.913773 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:23.913773 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:23.913773 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:23.913773 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:23.913773 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:23.913773 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:23.913773 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:23.913773 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:23.913773 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:23.913773 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:23.913773 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:23.914958 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:23.914958 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:23.914958 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:23.914958 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:23.914958 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 21:21:23.914958 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 21:21:23.914958 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 21:21:23.914958 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 21:21:23.914958 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 21:21:23.914958 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 21:21:23.914958 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 29 21:21:23.914958 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:21:23.914958 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 444/893] 10.222.0.90 () {56 vars in 1120 bytes} [Mon Aug 29 21:21:23 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 29 21:21:23.915662 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7a1bd5c-b702-499c-8d9f-31f5b93b645d None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:23.916048 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7a1bd5c-b702-499c-8d9f-31f5b93b645d None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:23.916150 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7a1bd5c-b702-499c-8d9f-31f5b93b645d None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:23.921032 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 445/894] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:21:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:21:23.922101 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7a1bd5c-b702-499c-8d9f-31f5b93b645d None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:23.980856 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9344cad-31d1-4800-9b33-3d357aea4599 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:24.020603 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae24e3e6-e511-44ef-a95f-2a38931edd80 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:24.021277 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae24e3e6-e511-44ef-a95f-2a38931edd80 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:24.021571 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae24e3e6-e511-44ef-a95f-2a38931edd80 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:24.021861 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae24e3e6-e511-44ef-a95f-2a38931edd80 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:24.022474 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae24e3e6-e511-44ef-a95f-2a38931edd80 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:24.023973 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae24e3e6-e511-44ef-a95f-2a38931edd80 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:24.025868 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:24.028159 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7a1bd5c-b702-499c-8d9f-31f5b93b645d None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:24.034995 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-ae24e3e6-e511-44ef-a95f-2a38931edd80 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 29 21:21:24.034995 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:24.034995 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 29 21:21:24.034995 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 29 21:21:24.034995 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 29 21:21:24.034995 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 29 21:21:24.034995 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 29 21:21:24.034995 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 21:21:24.034995 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 29 21:21:24.034995 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:21:24.034995 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 29 21:21:24.034995 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:21:24.034995 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:24.034995 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:24.034995 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:24.034995 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:24.035976 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:24.035976 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:24.035976 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:24.035976 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:24.035976 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:24.035976 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:24.035976 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:24.035976 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 29 21:21:24.035976 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 29 21:21:24.035976 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 29 21:21:24.035976 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 29 21:21:24.035976 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:24.035976 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:24.035976 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:24.035976 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:24.035976 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:24.035976 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:24.036879 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:24.036879 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:24.036879 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:24.036879 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:24.036879 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:24.036879 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:24.036879 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:24.036879 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:24.036879 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:24.036879 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:24.036879 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 29 21:21:24.036879 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 21:21:24.036879 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 29 21:21:24.036879 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:21:24.036879 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 450/895] 10.222.0.90 () {56 vars in 1132 bytes} [Mon Aug 29 21:21:23 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:21:24.037708 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38f6ce91-b69b-4cc2-805a-90d55d4d29bc None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:24.037708 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38f6ce91-b69b-4cc2-805a-90d55d4d29bc None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:24.038385 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38f6ce91-b69b-4cc2-805a-90d55d4d29bc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:24.057195 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-38f6ce91-b69b-4cc2-805a-90d55d4d29bc None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:21:24.058417 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-38f6ce91-b69b-4cc2-805a-90d55d4d29bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:21:24.063611 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:24.069714 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c225a91b-b657-4deb-ac44-7e48de017bdf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:24.070351 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c225a91b-b657-4deb-ac44-7e48de017bdf None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:24.070636 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c225a91b-b657-4deb-ac44-7e48de017bdf None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:24.070904 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c225a91b-b657-4deb-ac44-7e48de017bdf None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:24.071565 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c225a91b-b657-4deb-ac44-7e48de017bdf None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:24.071862 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c225a91b-b657-4deb-ac44-7e48de017bdf None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:24.073221 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c225a91b-b657-4deb-ac44-7e48de017bdf None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:24.081851 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 446/896] 10.222.0.90 () {56 vars in 1115 bytes} [Mon Aug 29 21:21:23 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:24.086807 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c225a91b-b657-4deb-ac44-7e48de017bdf None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:24.135990 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 451/897] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:21:24 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 21:21:24.137233 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38f6ce91-b69b-4cc2-805a-90d55d4d29bc None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:24.197565 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c225a91b-b657-4deb-ac44-7e48de017bdf None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:24.234288 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71939403-70fc-4362-a459-3778863aef2b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:24.234909 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71939403-70fc-4362-a459-3778863aef2b None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:24.235191 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71939403-70fc-4362-a459-3778863aef2b None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:24.235463 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71939403-70fc-4362-a459-3778863aef2b None admin] PATH_INFO: `/v3/services` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:24.236617 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71939403-70fc-4362-a459-3778863aef2b None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:24.237066 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71939403-70fc-4362-a459-3778863aef2b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:24.238477 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71939403-70fc-4362-a459-3778863aef2b None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:24.243384 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 447/898] 10.222.0.90 () {56 vars in 1126 bytes} [Mon Aug 29 21:21:24 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:24.249238 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71939403-70fc-4362-a459-3778863aef2b None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:24.273596 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38f6ce91-b69b-4cc2-805a-90d55d4d29bc None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:24.314156 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:24.315944 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:24.326618 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d01bbbb-d581-47d1-9fdd-99494eb7a416 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:24.327656 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d01bbbb-d581-47d1-9fdd-99494eb7a416 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:24.328278 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d01bbbb-d581-47d1-9fdd-99494eb7a416 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:24.328691 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d01bbbb-d581-47d1-9fdd-99494eb7a416 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:24.342667 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d01bbbb-d581-47d1-9fdd-99494eb7a416 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:24.345737 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d01bbbb-d581-47d1-9fdd-99494eb7a416 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:24.363950 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71939403-70fc-4362-a459-3778863aef2b None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:24.367292 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-5d01bbbb-d581-47d1-9fdd-99494eb7a416 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:21:24.367292 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:24.367292 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:24.367292 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:24.367292 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:24.367292 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:24.367292 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:24.367292 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:24.367292 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:24.367292 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:24.367292 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:24.367292 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:24.367292 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 21:21:24.367292 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 21:21:24.367292 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 21:21:24.367292 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 21:21:24.368442 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:24.368442 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:24.368442 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:24.368442 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:24.368442 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:24.368442 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:24.368442 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:24.368442 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:24.368442 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:24.368442 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:24.368442 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:24.368442 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:24.368442 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:24.368442 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:24.368442 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:24.368442 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:24.368442 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 21:21:24.369596 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 21:21:24.369596 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:21:24.369596 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:21:24.376610 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 452/899] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:21:24 2022] GET /identity/v3/domains/Default => generated 87 bytes in 240 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:21:24.377727 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d01bbbb-d581-47d1-9fdd-99494eb7a416 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:24.408410 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a29911a4-e202-40b6-818f-3fdbdcdd83bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:24.409729 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a29911a4-e202-40b6-818f-3fdbdcdd83bb None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:24.410143 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a29911a4-e202-40b6-818f-3fdbdcdd83bb None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:24.410520 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a29911a4-e202-40b6-818f-3fdbdcdd83bb None admin] PATH_INFO: `/v3/services` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:24.411387 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a29911a4-e202-40b6-818f-3fdbdcdd83bb None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:24.411809 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a29911a4-e202-40b6-818f-3fdbdcdd83bb None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:24.413270 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a29911a4-e202-40b6-818f-3fdbdcdd83bb None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:24.424003 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 448/900] 10.222.0.90 () {56 vars in 1126 bytes} [Mon Aug 29 21:21:24 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:24.425315 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a29911a4-e202-40b6-818f-3fdbdcdd83bb None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:24.502100 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d01bbbb-d581-47d1-9fdd-99494eb7a416 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:24.534562 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a29911a4-e202-40b6-818f-3fdbdcdd83bb None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:24.538064 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bca4c987-06a5-4bcd-8cd8-fcf95ba1385e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:24.538746 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca4c987-06a5-4bcd-8cd8-fcf95ba1385e None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:24.539029 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca4c987-06a5-4bcd-8cd8-fcf95ba1385e None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:24.539305 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca4c987-06a5-4bcd-8cd8-fcf95ba1385e None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:24.539993 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bca4c987-06a5-4bcd-8cd8-fcf95ba1385e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:24.540304 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bca4c987-06a5-4bcd-8cd8-fcf95ba1385e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:24.542334 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bca4c987-06a5-4bcd-8cd8-fcf95ba1385e None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:24.557369 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 453/901] 10.222.0.90 () {56 vars in 1109 bytes} [Mon Aug 29 21:21:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:24.558198 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bca4c987-06a5-4bcd-8cd8-fcf95ba1385e None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:24.565544 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:24.580475 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1ca69f7-a381-42a2-bed5-69d3425e75f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:24.581096 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1ca69f7-a381-42a2-bed5-69d3425e75f0 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:24.581381 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1ca69f7-a381-42a2-bed5-69d3425e75f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:24.581719 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1ca69f7-a381-42a2-bed5-69d3425e75f0 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:24.582471 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1ca69f7-a381-42a2-bed5-69d3425e75f0 None admin] RBAC: Adding query filter params (interface=public, service_id=015f8ddb93ed4f5e8d852cbe4633c883, region_id=RegionOne) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:24.582782 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1ca69f7-a381-42a2-bed5-69d3425e75f0 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:24.584149 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1ca69f7-a381-42a2-bed5-69d3425e75f0 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:24.589578 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 449/902] 10.222.0.90 () {56 vars in 1251 bytes} [Mon Aug 29 21:21:24 2022] GET /identity/v3/endpoints?service_id=015f8ddb93ed4f5e8d852cbe4633c883&interface=public®ion_id=RegionOne => generated 194 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:24.594987 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1ca69f7-a381-42a2-bed5-69d3425e75f0 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:24.605620 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:24.672830 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bca4c987-06a5-4bcd-8cd8-fcf95ba1385e None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:24.699784 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1ca69f7-a381-42a2-bed5-69d3425e75f0 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:24.711795 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b038e7f-02c9-46a5-b69d-ad1c121b76dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:24.712682 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b038e7f-02c9-46a5-b69d-ad1c121b76dc None admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:24.712968 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b038e7f-02c9-46a5-b69d-ad1c121b76dc None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:24.713246 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b038e7f-02c9-46a5-b69d-ad1c121b76dc None admin] PATH_INFO: `/v3/services` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:24.713954 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b038e7f-02c9-46a5-b69d-ad1c121b76dc None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:24.715318 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b038e7f-02c9-46a5-b69d-ad1c121b76dc None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:24.731902 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 454/903] 10.222.0.90 () {60 vars in 1141 bytes} [Mon Aug 29 21:21:24 2022] POST /identity/v3/services => generated 241 bytes in 174 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:21:24.742891 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b038e7f-02c9-46a5-b69d-ad1c121b76dc None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:24.746287 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d6d6627-884f-40cd-9c4f-d67bb85f5dd8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:24.746945 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d6d6627-884f-40cd-9c4f-d67bb85f5dd8 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:24.747229 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d6d6627-884f-40cd-9c4f-d67bb85f5dd8 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:24.747505 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d6d6627-884f-40cd-9c4f-d67bb85f5dd8 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:24.757298 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d6d6627-884f-40cd-9c4f-d67bb85f5dd8 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:24.758879 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d6d6627-884f-40cd-9c4f-d67bb85f5dd8 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:24.774489 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-1d6d6627-884f-40cd-9c4f-d67bb85f5dd8 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 29 21:21:24.774489 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:24.774489 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:24.774489 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:24.774489 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:24.774489 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:24.774489 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:24.774489 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:24.774489 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:24.774489 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:24.774489 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:24.774489 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:24.774489 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 29 21:21:24.774489 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 29 21:21:24.774489 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 29 21:21:24.774489 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 29 21:21:24.775547 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:24.775547 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:24.775547 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 29 21:21:24.775547 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 21:21:24.775547 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 29 21:21:24.775547 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 21:21:24.775547 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:24.775547 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:24.775547 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:24.775547 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:24.775547 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:24.775547 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:24.775547 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:24.775547 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:24.775547 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:24.775547 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:24.775547 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:24.775547 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:24.776499 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:24.776499 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:24.776499 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 29 21:21:24.776499 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 29 21:21:24.776499 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 29 21:21:24.776499 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 29 21:21:24.776499 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 29 21:21:24.776499 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 29 21:21:24.776499 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 29 21:21:24.776499 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:21:24.776499 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 450/904] 10.222.0.90 () {56 vars in 1117 bytes} [Mon Aug 29 21:21:24 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:21:24.781316 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d6d6627-884f-40cd-9c4f-d67bb85f5dd8 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:24.848568 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:24.890731 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:24.896007 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b038e7f-02c9-46a5-b69d-ad1c121b76dc None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:24.929538 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d6d6627-884f-40cd-9c4f-d67bb85f5dd8 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:24.946262 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f32de113-cd62-4bdc-9b9b-53f665dd699c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:24.947015 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f32de113-cd62-4bdc-9b9b-53f665dd699c None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:24.947302 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f32de113-cd62-4bdc-9b9b-53f665dd699c None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:24.947736 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f32de113-cd62-4bdc-9b9b-53f665dd699c None admin] PATH_INFO: `/v3/services` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:24.948666 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f32de113-cd62-4bdc-9b9b-53f665dd699c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:24.950082 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f32de113-cd62-4bdc-9b9b-53f665dd699c None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:24.957935 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 455/905] 10.222.0.90 () {56 vars in 1087 bytes} [Mon Aug 29 21:21:24 2022] GET /identity/v3/services => generated 1497 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:21:24.974717 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3437df41-de71-4944-8844-b86f15cee6ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:24.975359 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3437df41-de71-4944-8844-b86f15cee6ed None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:24.975642 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3437df41-de71-4944-8844-b86f15cee6ed None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:24.975918 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3437df41-de71-4944-8844-b86f15cee6ed None admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:24.976678 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3437df41-de71-4944-8844-b86f15cee6ed None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:24.976986 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3437df41-de71-4944-8844-b86f15cee6ed None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:24.978500 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3437df41-de71-4944-8844-b86f15cee6ed None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:25.005519 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 451/906] 10.222.0.90 () {56 vars in 1149 bytes} [Mon Aug 29 21:21:24 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:25.011387 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f32de113-cd62-4bdc-9b9b-53f665dd699c None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:25.134202 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f32de113-cd62-4bdc-9b9b-53f665dd699c None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:25.170113 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a190fdef-54da-4673-ab9b-f0a5d3632c5c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:25.170731 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a190fdef-54da-4673-ab9b-f0a5d3632c5c None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:25.171012 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a190fdef-54da-4673-ab9b-f0a5d3632c5c None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:25.171284 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a190fdef-54da-4673-ab9b-f0a5d3632c5c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:25.205922 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:25.245345 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a190fdef-54da-4673-ab9b-f0a5d3632c5c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:25.246843 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a190fdef-54da-4673-ab9b-f0a5d3632c5c None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:25.266574 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59b9b719-1518-496b-a9fc-0b8afad7dbea None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:25.267887 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59b9b719-1518-496b-a9fc-0b8afad7dbea None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:25.268373 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59b9b719-1518-496b-a9fc-0b8afad7dbea None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:25.270217 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 452/907] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:21:25 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:21:25.275452 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f874dfe-eaa6-4031-a7b5-0de6a9091927 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:25.276003 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f874dfe-eaa6-4031-a7b5-0de6a9091927 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:25.276485 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f874dfe-eaa6-4031-a7b5-0de6a9091927 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:25.296093 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-2f874dfe-eaa6-4031-a7b5-0de6a9091927 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:21:25.297303 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-2f874dfe-eaa6-4031-a7b5-0de6a9091927 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:21:25.369823 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 453/908] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:21:25 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 29 21:21:25.381608 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 456/909] 10.222.0.90 () {58 vars in 1282 bytes} [Mon Aug 29 21:21:25 2022] GET /identity/v3/auth/tokens => generated 1474 bytes in 371 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 29 21:21:25.387721 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f874dfe-eaa6-4031-a7b5-0de6a9091927 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:25.419481 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:25.499223 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f874dfe-eaa6-4031-a7b5-0de6a9091927 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:25.534998 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38959e82-4244-4bea-8423-454ca980624f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:25.535635 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38959e82-4244-4bea-8423-454ca980624f None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:25.535911 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38959e82-4244-4bea-8423-454ca980624f None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:25.536180 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38959e82-4244-4bea-8423-454ca980624f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:25.546415 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38959e82-4244-4bea-8423-454ca980624f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:25.547851 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38959e82-4244-4bea-8423-454ca980624f None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:25.562525 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-38959e82-4244-4bea-8423-454ca980624f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:21:25.562525 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:25.562525 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:25.562525 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:25.562525 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:25.562525 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:25.562525 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:25.562525 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:25.562525 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:25.562525 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:25.562525 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:25.562525 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:25.562525 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 21:21:25.562525 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 21:21:25.562525 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 21:21:25.562525 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 21:21:25.563593 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:25.563593 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:25.563593 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:25.563593 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:25.563593 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:25.563593 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:25.563593 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:25.563593 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:25.563593 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:25.563593 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:25.563593 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:25.563593 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:25.563593 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:25.563593 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:25.563593 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:25.563593 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:25.563593 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 21:21:25.565161 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 21:21:25.565161 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:21:25.565161 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:21:25.565161 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 454/910] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:21:25 2022] GET /identity/v3/domains/Default => generated 87 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:21:25.571714 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a190fdef-54da-4673-ab9b-f0a5d3632c5c None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:25.635401 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:25.675643 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a190fdef-54da-4673-ab9b-f0a5d3632c5c None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:25.693992 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38959e82-4244-4bea-8423-454ca980624f None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:25.716844 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-565a89f1-8938-4725-aa32-a0fd43917978 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:25.717804 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-565a89f1-8938-4725-aa32-a0fd43917978 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:25.718199 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-565a89f1-8938-4725-aa32-a0fd43917978 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:25.718486 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-565a89f1-8938-4725-aa32-a0fd43917978 None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:25.719301 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-565a89f1-8938-4725-aa32-a0fd43917978 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:25.719617 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-565a89f1-8938-4725-aa32-a0fd43917978 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:25.722053 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-565a89f1-8938-4725-aa32-a0fd43917978 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:25.739364 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 457/911] 10.222.0.90 () {56 vars in 1109 bytes} [Mon Aug 29 21:21:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:25.740272 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-565a89f1-8938-4725-aa32-a0fd43917978 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:25.797834 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:25.821557 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38959e82-4244-4bea-8423-454ca980624f None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:25.852548 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-565a89f1-8938-4725-aa32-a0fd43917978 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:25.867456 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39caa1c9-3bfd-41bd-92ad-ce322d4b2196 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:25.868386 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39caa1c9-3bfd-41bd-92ad-ce322d4b2196 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:25.868668 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39caa1c9-3bfd-41bd-92ad-ce322d4b2196 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:25.868942 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39caa1c9-3bfd-41bd-92ad-ce322d4b2196 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:25.869542 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39caa1c9-3bfd-41bd-92ad-ce322d4b2196 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:25.871111 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39caa1c9-3bfd-41bd-92ad-ce322d4b2196 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:25.882265 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:25.883166 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-39caa1c9-3bfd-41bd-92ad-ce322d4b2196 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 29 21:21:25.883166 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:25.883166 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 29 21:21:25.883166 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 29 21:21:25.883166 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 29 21:21:25.883166 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 29 21:21:25.883166 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 29 21:21:25.883166 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 21:21:25.883166 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 29 21:21:25.883166 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:21:25.883166 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 29 21:21:25.883166 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:21:25.883166 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:25.883166 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:25.883166 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:25.883166 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:25.883166 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:25.884509 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:25.884509 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:25.884509 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:25.884509 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:25.884509 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:25.884509 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:25.884509 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 29 21:21:25.884509 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 29 21:21:25.884509 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 29 21:21:25.884509 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 29 21:21:25.884509 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:25.884509 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:25.884509 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:25.884509 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:25.884509 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:25.884509 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:25.885740 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:25.885740 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:25.885740 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:25.885740 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:25.885740 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:25.885740 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:25.885740 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:25.885740 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:25.885740 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:25.885740 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:25.885740 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 29 21:21:25.885740 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 21:21:25.885740 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 29 21:21:25.885740 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:21:25.887598 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 455/912] 10.222.0.90 () {56 vars in 1114 bytes} [Mon Aug 29 21:21:25 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:21:25.893637 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39caa1c9-3bfd-41bd-92ad-ce322d4b2196 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:25.904258 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6a9d9df-ab97-4e1f-80ec-a776cfd8be32 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:25.905544 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6a9d9df-ab97-4e1f-80ec-a776cfd8be32 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:25.906064 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6a9d9df-ab97-4e1f-80ec-a776cfd8be32 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:25.906848 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6a9d9df-ab97-4e1f-80ec-a776cfd8be32 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:25.916263 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6a9d9df-ab97-4e1f-80ec-a776cfd8be32 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:25.918431 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6a9d9df-ab97-4e1f-80ec-a776cfd8be32 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:25.934461 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-a6a9d9df-ab97-4e1f-80ec-a776cfd8be32 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 21:21:25.934461 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:25.934461 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:25.934461 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:25.934461 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:25.934461 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:25.934461 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:25.934461 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:25.934461 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:25.934461 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:25.934461 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:25.934461 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:25.934461 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 29 21:21:25.934461 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 29 21:21:25.934461 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 29 21:21:25.935581 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 29 21:21:25.935581 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:25.935581 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:25.935581 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:25.935581 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:25.935581 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:25.935581 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:25.935581 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:25.935581 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:25.935581 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:25.935581 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:25.935581 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:25.935581 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:25.935581 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:25.935581 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:25.935581 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:25.935581 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:25.936697 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 29 21:21:25.936697 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 29 21:21:25.936697 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 29 21:21:25.936697 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 29 21:21:25.936697 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 29 21:21:25.936697 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 29 21:21:25.936697 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 21:21:25.936697 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:21:25.939405 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 458/913] 10.222.0.90 () {56 vars in 1111 bytes} [Mon Aug 29 21:21:25 2022] GET /identity/v3/projects/service => generated 88 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:21:25.945626 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6a9d9df-ab97-4e1f-80ec-a776cfd8be32 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:26.022812 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39caa1c9-3bfd-41bd-92ad-ce322d4b2196 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:26.061525 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82c127f9-4cc3-4bf8-9a19-36fcf7e8bd1f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:26.062118 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82c127f9-4cc3-4bf8-9a19-36fcf7e8bd1f None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:26.062471 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82c127f9-4cc3-4bf8-9a19-36fcf7e8bd1f None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:26.062939 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82c127f9-4cc3-4bf8-9a19-36fcf7e8bd1f None admin] PATH_INFO: `/v3/services` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:26.063594 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82c127f9-4cc3-4bf8-9a19-36fcf7e8bd1f None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:26.064027 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82c127f9-4cc3-4bf8-9a19-36fcf7e8bd1f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:26.065576 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82c127f9-4cc3-4bf8-9a19-36fcf7e8bd1f None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:26.066064 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6a9d9df-ab97-4e1f-80ec-a776cfd8be32 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:26.075553 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 456/914] 10.222.0.90 () {56 vars in 1114 bytes} [Mon Aug 29 21:21:25 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:21:26.076771 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82c127f9-4cc3-4bf8-9a19-36fcf7e8bd1f None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:26.104466 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:26.110327 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eef195e8-6478-48a6-9125-8393e9ba6492 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:26.111823 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef195e8-6478-48a6-9125-8393e9ba6492 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:26.112287 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef195e8-6478-48a6-9125-8393e9ba6492 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:26.112749 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef195e8-6478-48a6-9125-8393e9ba6492 None admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:26.113633 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eef195e8-6478-48a6-9125-8393e9ba6492 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:26.114277 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eef195e8-6478-48a6-9125-8393e9ba6492 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:26.116714 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eef195e8-6478-48a6-9125-8393e9ba6492 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:26.138018 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 459/915] 10.222.0.90 () {56 vars in 1148 bytes} [Mon Aug 29 21:21:25 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:21:26.139141 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eef195e8-6478-48a6-9125-8393e9ba6492 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:26.189943 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:26.201175 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82c127f9-4cc3-4bf8-9a19-36fcf7e8bd1f None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:26.240298 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13601c86-c1ea-4a3b-90c0-12ab4b78ee0a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:26.241341 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13601c86-c1ea-4a3b-90c0-12ab4b78ee0a None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:26.241860 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13601c86-c1ea-4a3b-90c0-12ab4b78ee0a None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:26.242348 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13601c86-c1ea-4a3b-90c0-12ab4b78ee0a None admin] PATH_INFO: `/v3/services` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:26.243280 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13601c86-c1ea-4a3b-90c0-12ab4b78ee0a None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:26.243881 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13601c86-c1ea-4a3b-90c0-12ab4b78ee0a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:26.245443 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13601c86-c1ea-4a3b-90c0-12ab4b78ee0a None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:26.249735 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eef195e8-6478-48a6-9125-8393e9ba6492 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:26.251635 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 457/916] 10.222.0.90 () {56 vars in 1114 bytes} [Mon Aug 29 21:21:26 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:26.257456 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13601c86-c1ea-4a3b-90c0-12ab4b78ee0a None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:26.293912 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89e88477-8d77-4bde-895f-a8822ba4949a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:26.294771 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89e88477-8d77-4bde-895f-a8822ba4949a None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:26.295153 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89e88477-8d77-4bde-895f-a8822ba4949a None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:26.295528 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89e88477-8d77-4bde-895f-a8822ba4949a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:26.296358 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89e88477-8d77-4bde-895f-a8822ba4949a None admin] RBAC: Adding query filter params (role.id=9c919ca974684bc492e75274b81c78ad, scope.project.id=ec03e112e3d547fbab6faf7b485c3fca, user.id=0c0b0da87b124de1be00faec4bbdb404) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:26.296789 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89e88477-8d77-4bde-895f-a8822ba4949a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:26.298670 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89e88477-8d77-4bde-895f-a8822ba4949a None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:26.313834 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 460/917] 10.222.0.90 () {56 vars in 1374 bytes} [Mon Aug 29 21:21:26 2022] GET /identity/v3/role_assignments?user.id=0c0b0da87b124de1be00faec4bbdb404&scope.project.id=ec03e112e3d547fbab6faf7b485c3fca&role.id=9c919ca974684bc492e75274b81c78ad => generated 259 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:26.355834 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:26.385835 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13601c86-c1ea-4a3b-90c0-12ab4b78ee0a None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:26.426434 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dcf5fa9-3191-4644-bf09-420ac0e3412e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:26.427115 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dcf5fa9-3191-4644-bf09-420ac0e3412e None admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:26.427401 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dcf5fa9-3191-4644-bf09-420ac0e3412e None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:26.427678 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dcf5fa9-3191-4644-bf09-420ac0e3412e None admin] PATH_INFO: `/v3/endpoints` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:26.428379 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dcf5fa9-3191-4644-bf09-420ac0e3412e None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:26.429778 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dcf5fa9-3191-4644-bf09-420ac0e3412e None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:26.456291 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 458/918] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:21:26 2022] POST /identity/v3/endpoints => generated 346 bytes in 199 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 21:21:26.803333 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6de16a3-a5ae-4a7c-92b9-353719a44646 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:26.803878 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6de16a3-a5ae-4a7c-92b9-353719a44646 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:26.803878 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6de16a3-a5ae-4a7c-92b9-353719a44646 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:26.805423 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 461/919] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:21:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:21:26.810829 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-086359db-bdec-4346-be90-6509a6d87a8b None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:26.811251 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-086359db-bdec-4346-be90-6509a6d87a8b None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:26.811422 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-086359db-bdec-4346-be90-6509a6d87a8b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:26.831502 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-086359db-bdec-4346-be90-6509a6d87a8b None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:21:26.832709 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-086359db-bdec-4346-be90-6509a6d87a8b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:21:26.914989 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 459/920] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:21:26 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 21:21:27.055569 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09a9e12e-7549-4224-83dc-1615006ddf70 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:27.056152 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09a9e12e-7549-4224-83dc-1615006ddf70 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:27.056615 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09a9e12e-7549-4224-83dc-1615006ddf70 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:27.058427 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 462/921] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:21:27 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:21:27.066944 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-724cb72f-9dd3-488c-9996-1f336e3520af None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:27.067485 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-724cb72f-9dd3-488c-9996-1f336e3520af None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:27.067978 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-724cb72f-9dd3-488c-9996-1f336e3520af None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:27.086870 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-724cb72f-9dd3-488c-9996-1f336e3520af None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:21:27.088345 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-724cb72f-9dd3-488c-9996-1f336e3520af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:21:27.142006 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:27.161395 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 460/922] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:21:27 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 21:21:27.188071 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09a9e12e-7549-4224-83dc-1615006ddf70 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:27.297549 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09a9e12e-7549-4224-83dc-1615006ddf70 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:27.310218 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:27.335751 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e3a6041-092a-47ba-b208-052953323242 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:27.336399 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e3a6041-092a-47ba-b208-052953323242 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:27.336687 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e3a6041-092a-47ba-b208-052953323242 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:27.337009 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e3a6041-092a-47ba-b208-052953323242 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:27.337948 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e3a6041-092a-47ba-b208-052953323242 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:27.339483 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e3a6041-092a-47ba-b208-052953323242 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:27.349936 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-7e3a6041-092a-47ba-b208-052953323242 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 29 21:21:27.349936 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:27.349936 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 29 21:21:27.349936 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 29 21:21:27.349936 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 29 21:21:27.349936 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 29 21:21:27.349936 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 29 21:21:27.349936 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 21:21:27.349936 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 29 21:21:27.349936 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:21:27.349936 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 29 21:21:27.349936 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:21:27.349936 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:27.349936 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:27.349936 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:27.349936 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:27.349936 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:27.350982 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:27.350982 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:27.350982 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:27.350982 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:27.350982 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:27.350982 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:27.350982 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 29 21:21:27.350982 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 29 21:21:27.350982 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 29 21:21:27.350982 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 29 21:21:27.350982 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:27.350982 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:27.350982 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:27.350982 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:27.350982 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:27.350982 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:27.351824 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:27.351824 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:27.351824 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:27.351824 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:27.351824 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:27.351824 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:27.351824 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:27.351824 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:27.351824 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:27.351824 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:27.351824 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 29 21:21:27.351824 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 21:21:27.351824 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 29 21:21:27.351824 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:21:27.351824 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 463/923] 10.222.0.90 () {56 vars in 1126 bytes} [Mon Aug 29 21:21:27 2022] GET /identity/v3/services/object-store => generated 93 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:21:27.357493 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-724cb72f-9dd3-488c-9996-1f336e3520af None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:27.433446 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e3a6041-092a-47ba-b208-052953323242 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:27.470598 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-724cb72f-9dd3-488c-9996-1f336e3520af None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:27.507988 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83116865-527a-4f3c-8728-299f75aa23d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:27.508611 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83116865-527a-4f3c-8728-299f75aa23d7 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:27.508894 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83116865-527a-4f3c-8728-299f75aa23d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:27.509170 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83116865-527a-4f3c-8728-299f75aa23d7 None admin] PATH_INFO: `/v3/services` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:27.510133 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83116865-527a-4f3c-8728-299f75aa23d7 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:27.510499 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83116865-527a-4f3c-8728-299f75aa23d7 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:27.511942 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83116865-527a-4f3c-8728-299f75aa23d7 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:27.517927 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 461/924] 10.222.0.90 () {56 vars in 1122 bytes} [Mon Aug 29 21:21:27 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:21:27.523501 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83116865-527a-4f3c-8728-299f75aa23d7 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:27.547679 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e3a6041-092a-47ba-b208-052953323242 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:27.584686 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcbf98c4-fbc9-4ceb-b2ed-96ec99001a45 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:27.585243 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcbf98c4-fbc9-4ceb-b2ed-96ec99001a45 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:27.585521 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcbf98c4-fbc9-4ceb-b2ed-96ec99001a45 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:27.585808 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcbf98c4-fbc9-4ceb-b2ed-96ec99001a45 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:27.586422 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcbf98c4-fbc9-4ceb-b2ed-96ec99001a45 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:27.587984 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcbf98c4-fbc9-4ceb-b2ed-96ec99001a45 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:27.599515 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-fcbf98c4-fbc9-4ceb-b2ed-96ec99001a45 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 29 21:21:27.599515 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:27.599515 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 29 21:21:27.599515 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 29 21:21:27.599515 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 29 21:21:27.599515 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 29 21:21:27.599515 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 29 21:21:27.599515 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 21:21:27.599515 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 29 21:21:27.599515 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:21:27.599515 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 29 21:21:27.599515 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:21:27.599515 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:27.599515 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:27.599515 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:27.599515 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:27.600541 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:27.600541 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:27.600541 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:27.600541 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:27.600541 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:27.600541 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:27.600541 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:27.600541 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 29 21:21:27.600541 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 29 21:21:27.600541 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 29 21:21:27.600541 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 29 21:21:27.600541 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:27.600541 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:27.600541 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:27.600541 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:27.600541 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:27.600541 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:27.601462 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:27.601462 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:27.601462 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:27.601462 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:27.601462 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:27.601462 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:27.601462 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:27.601462 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:27.601462 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:27.601462 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:27.601462 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 29 21:21:27.601462 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 21:21:27.601462 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 29 21:21:27.601462 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:21:27.603802 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:27.605194 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 464/925] 10.222.0.90 () {56 vars in 1132 bytes} [Mon Aug 29 21:21:27 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 29 21:21:27.607172 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcbf98c4-fbc9-4ceb-b2ed-96ec99001a45 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:27.651011 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:27.674151 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83116865-527a-4f3c-8728-299f75aa23d7 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:27.719011 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c384035-f4c4-4c04-b11b-f60ab30b84c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:27.720057 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c384035-f4c4-4c04-b11b-f60ab30b84c5 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:27.720057 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c384035-f4c4-4c04-b11b-f60ab30b84c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:27.720361 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c384035-f4c4-4c04-b11b-f60ab30b84c5 None admin] PATH_INFO: `/v3/services` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:27.721411 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c384035-f4c4-4c04-b11b-f60ab30b84c5 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:27.721814 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c384035-f4c4-4c04-b11b-f60ab30b84c5 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:27.723273 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c384035-f4c4-4c04-b11b-f60ab30b84c5 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:27.728802 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 462/926] 10.222.0.90 () {56 vars in 1122 bytes} [Mon Aug 29 21:21:27 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:21:27.735089 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c384035-f4c4-4c04-b11b-f60ab30b84c5 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:27.741509 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcbf98c4-fbc9-4ceb-b2ed-96ec99001a45 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:27.787276 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8cd685d-8267-4931-b29d-27092ae1ca3c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:27.788328 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8cd685d-8267-4931-b29d-27092ae1ca3c None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:27.789063 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8cd685d-8267-4931-b29d-27092ae1ca3c None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:27.789557 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8cd685d-8267-4931-b29d-27092ae1ca3c None admin] PATH_INFO: `/v3/services` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:27.790523 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8cd685d-8267-4931-b29d-27092ae1ca3c None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:27.791122 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8cd685d-8267-4931-b29d-27092ae1ca3c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:27.793270 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8cd685d-8267-4931-b29d-27092ae1ca3c None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:27.801333 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 465/927] 10.222.0.90 () {56 vars in 1126 bytes} [Mon Aug 29 21:21:27 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:21:27.807585 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8cd685d-8267-4931-b29d-27092ae1ca3c None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:27.848907 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c384035-f4c4-4c04-b11b-f60ab30b84c5 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:27.864469 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:27.888417 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e58dc783-ed24-4786-a0dc-44429736d74b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:27.889034 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e58dc783-ed24-4786-a0dc-44429736d74b None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:27.889328 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e58dc783-ed24-4786-a0dc-44429736d74b None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:27.889831 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e58dc783-ed24-4786-a0dc-44429736d74b None admin] PATH_INFO: `/v3/endpoints` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:27.891017 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e58dc783-ed24-4786-a0dc-44429736d74b None admin] RBAC: Adding query filter params (interface=public, service_id=c69fc69e24984d4191af3077686e5ca3, region_id=RegionOne) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:27.891339 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e58dc783-ed24-4786-a0dc-44429736d74b None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:27.893997 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e58dc783-ed24-4786-a0dc-44429736d74b None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:27.911432 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 463/928] 10.222.0.90 () {56 vars in 1251 bytes} [Mon Aug 29 21:21:27 2022] GET /identity/v3/endpoints?service_id=c69fc69e24984d4191af3077686e5ca3&interface=public®ion_id=RegionOne => generated 194 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:27.912589 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e58dc783-ed24-4786-a0dc-44429736d74b None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:27.935464 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:27.939975 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8cd685d-8267-4931-b29d-27092ae1ca3c None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:27.976581 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49b9a71b-8d13-4575-b7f7-22a97c7ab478 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:27.977574 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b9a71b-8d13-4575-b7f7-22a97c7ab478 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:27.978120 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b9a71b-8d13-4575-b7f7-22a97c7ab478 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:27.978601 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b9a71b-8d13-4575-b7f7-22a97c7ab478 None admin] PATH_INFO: `/v3/services` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:27.979442 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49b9a71b-8d13-4575-b7f7-22a97c7ab478 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:27.979950 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49b9a71b-8d13-4575-b7f7-22a97c7ab478 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:27.981538 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49b9a71b-8d13-4575-b7f7-22a97c7ab478 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:27.988174 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 466/929] 10.222.0.90 () {56 vars in 1126 bytes} [Mon Aug 29 21:21:27 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:27.993653 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49b9a71b-8d13-4575-b7f7-22a97c7ab478 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:28.035544 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e58dc783-ed24-4786-a0dc-44429736d74b None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:28.080973 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3b6a5b1-2b72-42ac-950b-c682e18aa886 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:28.082619 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3b6a5b1-2b72-42ac-950b-c682e18aa886 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:28.083532 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3b6a5b1-2b72-42ac-950b-c682e18aa886 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:28.084016 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3b6a5b1-2b72-42ac-950b-c682e18aa886 None admin] PATH_INFO: `/v3/services` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:28.085107 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3b6a5b1-2b72-42ac-950b-c682e18aa886 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:28.087646 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3b6a5b1-2b72-42ac-950b-c682e18aa886 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:28.102299 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 464/930] 10.222.0.90 () {56 vars in 1087 bytes} [Mon Aug 29 21:21:27 2022] GET /identity/v3/services => generated 1497 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:21:28.124955 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49b9a71b-8d13-4575-b7f7-22a97c7ab478 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:28.165137 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaf11727-188e-40a9-8c48-24ea82fdaf0a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:28.165864 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf11727-188e-40a9-8c48-24ea82fdaf0a None admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:28.166152 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf11727-188e-40a9-8c48-24ea82fdaf0a None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:28.166427 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf11727-188e-40a9-8c48-24ea82fdaf0a None admin] PATH_INFO: `/v3/endpoints` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:28.167116 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaf11727-188e-40a9-8c48-24ea82fdaf0a None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:28.168529 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaf11727-188e-40a9-8c48-24ea82fdaf0a None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:28.197404 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 467/931] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:21:27 2022] POST /identity/v3/endpoints => generated 347 bytes in 204 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 21:21:28.317082 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd5573c3-8554-4253-be61-9e56116c6cf3 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:28.317873 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd5573c3-8554-4253-be61-9e56116c6cf3 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:28.318480 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd5573c3-8554-4253-be61-9e56116c6cf3 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:28.320400 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 465/932] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:21:28 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:21:28.326208 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e93dab07-d82a-49b8-b70a-66d3c82a831e None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:28.326788 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e93dab07-d82a-49b8-b70a-66d3c82a831e None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:28.327298 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e93dab07-d82a-49b8-b70a-66d3c82a831e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:28.349547 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-e93dab07-d82a-49b8-b70a-66d3c82a831e None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:21:28.351111 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-e93dab07-d82a-49b8-b70a-66d3c82a831e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:21:28.367404 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:28.431940 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 468/933] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:21:28 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 21:21:28.539148 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ed03488-f12d-4e7e-9313-4396edde62f0 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:28.539512 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ed03488-f12d-4e7e-9313-4396edde62f0 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:28.539686 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ed03488-f12d-4e7e-9313-4396edde62f0 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:28.541133 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 466/934] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:21:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:21:28.546860 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adc7e434-bc46-4471-aebb-e7a4e5fa21d1 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:28.547402 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adc7e434-bc46-4471-aebb-e7a4e5fa21d1 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:28.547903 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adc7e434-bc46-4471-aebb-e7a4e5fa21d1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:28.566911 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-adc7e434-bc46-4471-aebb-e7a4e5fa21d1 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:21:28.568429 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-adc7e434-bc46-4471-aebb-e7a4e5fa21d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:21:28.642079 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 469/935] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:21:28 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 21:21:28.709329 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ed03488-f12d-4e7e-9313-4396edde62f0 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:28.757478 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:28.817581 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ed03488-f12d-4e7e-9313-4396edde62f0 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:28.852896 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b032b105-fb06-4863-a3fa-4cfbf17897c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:28.853534 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b032b105-fb06-4863-a3fa-4cfbf17897c0 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:28.854272 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b032b105-fb06-4863-a3fa-4cfbf17897c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:28.854549 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b032b105-fb06-4863-a3fa-4cfbf17897c0 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:28.865082 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b032b105-fb06-4863-a3fa-4cfbf17897c0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 21:21:28.869868 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-b032b105-fb06-4863-a3fa-4cfbf17897c0 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 29 21:21:28.869868 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:28.869868 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:28.869868 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:28.869868 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:28.869868 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:28.869868 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:28.869868 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:28.869868 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:28.869868 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:28.869868 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:28.869868 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:28.869868 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 21:21:28.869868 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 21:21:28.869868 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 21:21:28.869868 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 21:21:28.870833 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 21:21:28.870833 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 21:21:28.870833 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 21:21:28.870833 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 21:21:28.870833 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:28.870833 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:28.870833 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:28.870833 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:28.870833 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:28.870833 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:28.870833 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:28.870833 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:28.870833 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:28.870833 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:28.870833 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:28.870833 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:28.871692 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:28.871692 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:28.871692 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:28.871692 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:28.871692 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 21:21:28.871692 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 21:21:28.871692 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 21:21:28.871692 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 21:21:28.871692 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 21:21:28.871692 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 21:21:28.871692 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 29 21:21:28.871692 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:21:28.871692 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 467/936] 10.222.0.90 () {56 vars in 1120 bytes} [Mon Aug 29 21:21:28 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:21:28.877959 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adc7e434-bc46-4471-aebb-e7a4e5fa21d1 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:28.927570 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b032b105-fb06-4863-a3fa-4cfbf17897c0 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:28.969102 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:29.007353 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adc7e434-bc46-4471-aebb-e7a4e5fa21d1 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:29.047879 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b032b105-fb06-4863-a3fa-4cfbf17897c0 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:29.050102 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:29.059968 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31a0a2e5-4b6b-4e9d-a8c2-23fdfe4d4aa8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:29.061353 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31a0a2e5-4b6b-4e9d-a8c2-23fdfe4d4aa8 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:29.062311 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31a0a2e5-4b6b-4e9d-a8c2-23fdfe4d4aa8 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:29.062951 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31a0a2e5-4b6b-4e9d-a8c2-23fdfe4d4aa8 None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:29.064568 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31a0a2e5-4b6b-4e9d-a8c2-23fdfe4d4aa8 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:29.065109 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31a0a2e5-4b6b-4e9d-a8c2-23fdfe4d4aa8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:29.067533 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31a0a2e5-4b6b-4e9d-a8c2-23fdfe4d4aa8 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:29.080378 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 470/937] 10.222.0.90 () {56 vars in 1115 bytes} [Mon Aug 29 21:21:28 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:29.086171 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31a0a2e5-4b6b-4e9d-a8c2-23fdfe4d4aa8 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:29.089506 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a670c0ce-7501-441a-aae2-cbb026c0fd2f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:29.090263 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a670c0ce-7501-441a-aae2-cbb026c0fd2f None admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:29.090791 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a670c0ce-7501-441a-aae2-cbb026c0fd2f None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:29.091326 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a670c0ce-7501-441a-aae2-cbb026c0fd2f None admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:29.092169 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a670c0ce-7501-441a-aae2-cbb026c0fd2f None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:29.093843 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a670c0ce-7501-441a-aae2-cbb026c0fd2f None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:29.097206 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: WARNING keystone.server.flask.common [None req-a670c0ce-7501-441a-aae2-cbb026c0fd2f None admin] Deprecated: Not specifying a domain during a create user, group or project call, and relying on falling back to the default domain, is deprecated as of Liberty. There is no plan to remove this compatibility, however, future API versions may remove this, so please specify the domain explicitly or use a domain-scoped token. Aug 29 21:21:29.115996 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.sql.core [None req-a670c0ce-7501-441a-aae2-cbb026c0fd2f None admin] Conflict project: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Aug 29 21:21:29.115996 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Aug 29 21:21:29.115996 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [parameters: {'id': '8e6e994ee0da4908a1f9b63ab5fd4d14', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Aug 29 21:21:29.115996 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81434) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Aug 29 21:21:29.127306 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-a670c0ce-7501-441a-aae2-cbb026c0fd2f None admin] Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 8e6e994ee0da4908a1f9b63ab5fd4d14.: keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 8e6e994ee0da4908a1f9b63ab5fd4d14. Aug 29 21:21:29.127306 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:29.127306 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Aug 29 21:21:29.127306 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application self.dialect.do_execute( Aug 29 21:21:29.127306 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Aug 29 21:21:29.127306 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Aug 29 21:21:29.127306 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Aug 29 21:21:29.127306 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application result = self._query(query) Aug 29 21:21:29.127306 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Aug 29 21:21:29.127306 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application conn.query(q) Aug 29 21:21:29.127306 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Aug 29 21:21:29.127306 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Aug 29 21:21:29.127306 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Aug 29 21:21:29.128159 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application result.read() Aug 29 21:21:29.128159 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Aug 29 21:21:29.128159 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Aug 29 21:21:29.128159 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Aug 29 21:21:29.128159 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application packet.raise_for_error() Aug 29 21:21:29.128159 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Aug 29 21:21:29.128159 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Aug 29 21:21:29.128159 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Aug 29 21:21:29.128159 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Aug 29 21:21:29.128159 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Aug 29 21:21:29.128159 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:21:29.128159 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Aug 29 21:21:29.128159 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:21:29.128159 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:29.128159 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Aug 29 21:21:29.128159 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return method(*args, **kwargs) Aug 29 21:21:29.128159 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 241, in create_project Aug 29 21:21:29.128159 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return project_ref.to_dict() Aug 29 21:21:29.129134 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Aug 29 21:21:29.129134 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application next(self.gen) Aug 29 21:21:29.129134 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Aug 29 21:21:29.129134 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application yield resource Aug 29 21:21:29.129134 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Aug 29 21:21:29.129134 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application next(self.gen) Aug 29 21:21:29.129134 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Aug 29 21:21:29.129134 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application self.session.rollback() Aug 29 21:21:29.129134 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Aug 29 21:21:29.129134 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application self.force_reraise() Aug 29 21:21:29.129134 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Aug 29 21:21:29.129134 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise self.value Aug 29 21:21:29.129134 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Aug 29 21:21:29.129134 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Aug 29 21:21:29.129134 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Aug 29 21:21:29.129134 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application session.commit() Aug 29 21:21:29.129134 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Aug 29 21:21:29.130074 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Aug 29 21:21:29.130074 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Aug 29 21:21:29.130074 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application self._prepare_impl() Aug 29 21:21:29.130074 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Aug 29 21:21:29.130074 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application self.session.flush() Aug 29 21:21:29.130074 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Aug 29 21:21:29.130074 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application self._flush(objects) Aug 29 21:21:29.130074 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Aug 29 21:21:29.130074 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Aug 29 21:21:29.130074 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Aug 29 21:21:29.130074 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application compat.raise_( Aug 29 21:21:29.130074 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Aug 29 21:21:29.130074 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception Aug 29 21:21:29.130074 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Aug 29 21:21:29.130074 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application flush_context.execute() Aug 29 21:21:29.130074 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Aug 29 21:21:29.130074 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rec.execute(self) Aug 29 21:21:29.130973 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Aug 29 21:21:29.130973 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Aug 29 21:21:29.130973 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Aug 29 21:21:29.130973 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application _emit_insert_statements( Aug 29 21:21:29.130973 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Aug 29 21:21:29.130973 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application c = connection._execute_20( Aug 29 21:21:29.130973 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Aug 29 21:21:29.130973 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Aug 29 21:21:29.130973 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Aug 29 21:21:29.130973 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Aug 29 21:21:29.130973 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Aug 29 21:21:29.130973 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ret = self._execute_context( Aug 29 21:21:29.130973 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Aug 29 21:21:29.130973 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Aug 29 21:21:29.130973 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Aug 29 21:21:29.131765 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Aug 29 21:21:29.131765 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Aug 29 21:21:29.131765 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception Aug 29 21:21:29.131765 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Aug 29 21:21:29.131765 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application self.dialect.do_execute( Aug 29 21:21:29.131765 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Aug 29 21:21:29.131765 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Aug 29 21:21:29.131765 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Aug 29 21:21:29.131765 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application result = self._query(query) Aug 29 21:21:29.131765 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Aug 29 21:21:29.131765 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application conn.query(q) Aug 29 21:21:29.131765 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Aug 29 21:21:29.131765 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Aug 29 21:21:29.131765 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Aug 29 21:21:29.131765 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application result.read() Aug 29 21:21:29.131765 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Aug 29 21:21:29.131765 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Aug 29 21:21:29.132661 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Aug 29 21:21:29.132661 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application packet.raise_for_error() Aug 29 21:21:29.132661 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Aug 29 21:21:29.132661 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Aug 29 21:21:29.132661 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Aug 29 21:21:29.132661 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Aug 29 21:21:29.132661 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Aug 29 21:21:29.132661 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Aug 29 21:21:29.132661 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [parameters: {'id': '8e6e994ee0da4908a1f9b63ab5fd4d14', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Aug 29 21:21:29.132661 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Aug 29 21:21:29.132661 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:21:29.132661 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 29 21:21:29.132661 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:21:29.132661 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:29.132661 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 216, in create_project Aug 29 21:21:29.135313 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ret = self.driver.create_project(project_id, project) Aug 29 21:21:29.135313 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Aug 29 21:21:29.135313 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Aug 29 21:21:29.135313 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - Duplicate entry found with name service at domain ID default. Aug 29 21:21:29.135313 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:21:29.135313 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 29 21:21:29.135313 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:21:29.135313 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:29.135313 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:29.135313 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:29.135313 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:29.135313 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:29.135313 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:29.135313 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:29.135313 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:29.135313 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:29.135313 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:29.138664 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:29.138664 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 176, in post Aug 29 21:21:29.138664 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ref = PROVIDERS.resource_api.create_project( Aug 29 21:21:29.138664 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:29.138664 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:29.138664 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 218, in create_project Aug 29 21:21:29.138664 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.Conflict( Aug 29 21:21:29.138664 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 8e6e994ee0da4908a1f9b63ab5fd4d14. Aug 29 21:21:29.138664 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:21:29.138664 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 468/938] 10.222.0.90 () {60 vars in 1140 bytes} [Mon Aug 29 21:21:28 2022] POST /identity/v3/projects => generated 262 bytes in 204 msecs (HTTP/1.1 409) 5 headers in 183 bytes (2 switches on core 0) Aug 29 21:21:29.140251 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a670c0ce-7501-441a-aae2-cbb026c0fd2f None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:29.213988 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31a0a2e5-4b6b-4e9d-a8c2-23fdfe4d4aa8 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:29.259668 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47fb28bc-63b2-4b82-8c30-6c4bc4ee1949 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:29.260856 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47fb28bc-63b2-4b82-8c30-6c4bc4ee1949 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:29.261642 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47fb28bc-63b2-4b82-8c30-6c4bc4ee1949 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:29.262390 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47fb28bc-63b2-4b82-8c30-6c4bc4ee1949 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:29.267904 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a670c0ce-7501-441a-aae2-cbb026c0fd2f None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:29.272538 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:29.276632 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47fb28bc-63b2-4b82-8c30-6c4bc4ee1949 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:29.278929 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47fb28bc-63b2-4b82-8c30-6c4bc4ee1949 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:29.297475 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-47fb28bc-63b2-4b82-8c30-6c4bc4ee1949 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:21:29.297475 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:29.297475 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:29.297475 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:29.297475 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:29.297475 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:29.297475 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:29.297475 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:29.297475 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:29.297475 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:29.297475 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:29.297475 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:29.297475 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 21:21:29.297475 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 21:21:29.297475 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 21:21:29.297475 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 21:21:29.299061 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:29.299061 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:29.299061 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:29.299061 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:29.299061 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:29.299061 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:29.299061 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:29.299061 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:29.299061 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:29.299061 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:29.299061 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:29.299061 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:29.299061 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:29.299061 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:29.299061 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:29.299061 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:29.299061 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 21:21:29.299989 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 21:21:29.299989 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:21:29.299989 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:21:29.306625 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 471/939] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:21:29 2022] GET /identity/v3/domains/Default => generated 87 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:21:29.307627 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47fb28bc-63b2-4b82-8c30-6c4bc4ee1949 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:29.316372 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7892bc95-83cf-4422-8a26-6fa0a5f01c00 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:29.317029 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7892bc95-83cf-4422-8a26-6fa0a5f01c00 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:29.317311 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7892bc95-83cf-4422-8a26-6fa0a5f01c00 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:29.317576 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7892bc95-83cf-4422-8a26-6fa0a5f01c00 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:29.325314 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7892bc95-83cf-4422-8a26-6fa0a5f01c00 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:29.327105 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7892bc95-83cf-4422-8a26-6fa0a5f01c00 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:29.340614 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-7892bc95-83cf-4422-8a26-6fa0a5f01c00 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 21:21:29.340614 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:29.340614 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:29.340614 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:29.340614 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:29.340614 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:29.340614 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:29.340614 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:29.340614 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:29.340614 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:29.340614 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:29.340614 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:29.340614 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 29 21:21:29.340614 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 29 21:21:29.340614 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 29 21:21:29.341622 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 29 21:21:29.341622 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:29.341622 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:29.341622 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:29.341622 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:29.341622 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:29.341622 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:29.341622 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:29.341622 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:29.341622 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:29.341622 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:29.341622 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:29.341622 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:29.341622 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:29.341622 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:29.341622 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:29.341622 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:29.342568 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 29 21:21:29.342568 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 29 21:21:29.342568 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 29 21:21:29.342568 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 29 21:21:29.342568 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 29 21:21:29.342568 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 29 21:21:29.342568 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 21:21:29.342568 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:21:29.346402 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 469/940] 10.222.0.90 () {56 vars in 1111 bytes} [Mon Aug 29 21:21:29 2022] GET /identity/v3/projects/service => generated 88 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:21:29.349016 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7892bc95-83cf-4422-8a26-6fa0a5f01c00 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:29.372151 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:29.450085 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47fb28bc-63b2-4b82-8c30-6c4bc4ee1949 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:29.480688 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7892bc95-83cf-4422-8a26-6fa0a5f01c00 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:29.490018 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e8a6ec7-cac2-4105-8d1c-199bffa6cf2d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:29.490857 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e8a6ec7-cac2-4105-8d1c-199bffa6cf2d None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:29.491289 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e8a6ec7-cac2-4105-8d1c-199bffa6cf2d None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:29.491761 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e8a6ec7-cac2-4105-8d1c-199bffa6cf2d None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:29.492559 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e8a6ec7-cac2-4105-8d1c-199bffa6cf2d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:29.493011 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e8a6ec7-cac2-4105-8d1c-199bffa6cf2d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:29.494562 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e8a6ec7-cac2-4105-8d1c-199bffa6cf2d None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:29.512241 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 472/941] 10.222.0.90 () {56 vars in 1109 bytes} [Mon Aug 29 21:21:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:29.513373 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e8a6ec7-cac2-4105-8d1c-199bffa6cf2d None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:29.522350 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02105e79-153e-48ee-98de-45140148f36d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:29.523154 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02105e79-153e-48ee-98de-45140148f36d None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:29.523566 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02105e79-153e-48ee-98de-45140148f36d None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:29.524281 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02105e79-153e-48ee-98de-45140148f36d None admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:29.524793 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02105e79-153e-48ee-98de-45140148f36d None admin] RBAC: Adding query filter params (name=service) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:29.525218 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02105e79-153e-48ee-98de-45140148f36d None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:29.526923 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02105e79-153e-48ee-98de-45140148f36d None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:29.558229 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 470/942] 10.222.0.90 () {56 vars in 1112 bytes} [Mon Aug 29 21:21:29 2022] GET /identity/v3/projects?name=service => generated 411 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:21:29.571868 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:29.629841 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e8a6ec7-cac2-4105-8d1c-199bffa6cf2d None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:29.668574 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7843e688-0170-4c51-9678-e4cce060a8a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:29.669239 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7843e688-0170-4c51-9678-e4cce060a8a7 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:29.669519 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7843e688-0170-4c51-9678-e4cce060a8a7 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:29.669812 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7843e688-0170-4c51-9678-e4cce060a8a7 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:29.678829 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7843e688-0170-4c51-9678-e4cce060a8a7 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:29.680295 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7843e688-0170-4c51-9678-e4cce060a8a7 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:29.693754 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-7843e688-0170-4c51-9678-e4cce060a8a7 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 29 21:21:29.693754 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:29.693754 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:29.693754 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:29.693754 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:29.693754 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:29.693754 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:29.693754 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:29.693754 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:29.693754 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:29.693754 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:29.693754 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:29.693754 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 29 21:21:29.693754 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 29 21:21:29.693754 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 29 21:21:29.693754 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 29 21:21:29.694844 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:29.694844 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:29.694844 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 29 21:21:29.694844 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 21:21:29.694844 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 29 21:21:29.694844 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 21:21:29.694844 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:29.694844 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:29.694844 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:29.694844 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:29.694844 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:29.694844 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:29.694844 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:29.694844 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:29.694844 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:29.694844 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:29.694844 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:29.694844 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:29.695841 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:29.695841 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:29.695841 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 29 21:21:29.695841 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 29 21:21:29.695841 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 29 21:21:29.695841 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 29 21:21:29.695841 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 29 21:21:29.695841 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 29 21:21:29.695841 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 29 21:21:29.695841 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:21:29.695841 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 473/943] 10.222.0.90 () {56 vars in 1117 bytes} [Mon Aug 29 21:21:29 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:21:29.701096 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02105e79-153e-48ee-98de-45140148f36d None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:29.792948 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:29.817973 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02105e79-153e-48ee-98de-45140148f36d None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:29.854974 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bdda997-a423-4a97-9f2c-887917be1a41 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:29.855641 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bdda997-a423-4a97-9f2c-887917be1a41 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:29.855887 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bdda997-a423-4a97-9f2c-887917be1a41 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:29.856200 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bdda997-a423-4a97-9f2c-887917be1a41 None admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:29.856954 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bdda997-a423-4a97-9f2c-887917be1a41 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:29.857260 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bdda997-a423-4a97-9f2c-887917be1a41 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:29.858743 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bdda997-a423-4a97-9f2c-887917be1a41 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:29.879641 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 471/944] 10.222.0.90 () {56 vars in 1149 bytes} [Mon Aug 29 21:21:29 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:29.885518 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7843e688-0170-4c51-9678-e4cce060a8a7 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:29.999557 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7843e688-0170-4c51-9678-e4cce060a8a7 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:30.010541 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:30.036066 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e07fdb36-18aa-41bf-9d10-5858ce93cd47 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:30.036730 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e07fdb36-18aa-41bf-9d10-5858ce93cd47 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:30.036981 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e07fdb36-18aa-41bf-9d10-5858ce93cd47 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:30.037254 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e07fdb36-18aa-41bf-9d10-5858ce93cd47 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:30.048433 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e07fdb36-18aa-41bf-9d10-5858ce93cd47 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:30.049824 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e07fdb36-18aa-41bf-9d10-5858ce93cd47 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:30.064617 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-e07fdb36-18aa-41bf-9d10-5858ce93cd47 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:21:30.064617 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:30.064617 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:30.064617 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:30.064617 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:30.064617 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:30.064617 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:30.064617 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:30.064617 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:30.064617 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:30.064617 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:30.064617 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:30.064617 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 21:21:30.064617 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 21:21:30.064617 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 21:21:30.064617 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 21:21:30.065644 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:30.065644 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:30.065644 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:30.065644 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:30.065644 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:30.065644 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:30.065644 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:30.065644 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:30.065644 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:30.065644 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:30.065644 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:30.065644 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:30.065644 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:30.065644 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:30.065644 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:30.065644 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:30.065644 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 21:21:30.066637 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 21:21:30.066637 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:21:30.066637 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:21:30.066637 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 474/945] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:21:29 2022] GET /identity/v3/domains/Default => generated 87 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:21:30.073205 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bdda997-a423-4a97-9f2c-887917be1a41 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:30.161506 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5757557a-f0da-4377-a3fb-1295d412a897 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:30.162199 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5757557a-f0da-4377-a3fb-1295d412a897 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:30.162780 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5757557a-f0da-4377-a3fb-1295d412a897 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:30.164619 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 475/946] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:21:30 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:21:30.170376 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f24769c-6b4f-41bc-98ee-a68336df1088 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:30.170958 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f24769c-6b4f-41bc-98ee-a68336df1088 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:30.171471 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f24769c-6b4f-41bc-98ee-a68336df1088 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:30.180674 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bdda997-a423-4a97-9f2c-887917be1a41 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:30.191775 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-1f24769c-6b4f-41bc-98ee-a68336df1088 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:21:30.193421 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-1f24769c-6b4f-41bc-98ee-a68336df1088 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:21:30.216962 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f87c0b13-1b0e-44c0-8c46-c20da3454e0e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:30.217590 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f87c0b13-1b0e-44c0-8c46-c20da3454e0e None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:30.217954 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f87c0b13-1b0e-44c0-8c46-c20da3454e0e None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:30.218231 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f87c0b13-1b0e-44c0-8c46-c20da3454e0e None admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:30.218886 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f87c0b13-1b0e-44c0-8c46-c20da3454e0e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:30.219190 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f87c0b13-1b0e-44c0-8c46-c20da3454e0e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:30.221483 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f87c0b13-1b0e-44c0-8c46-c20da3454e0e None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:30.224171 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:30.237839 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 472/947] 10.222.0.90 () {56 vars in 1109 bytes} [Mon Aug 29 21:21:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:30.240625 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f87c0b13-1b0e-44c0-8c46-c20da3454e0e None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:30.278175 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 476/948] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:21:30 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 29 21:21:30.334427 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6e0e3af-5ca4-4e3f-9b7d-6b89b07cf56f None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:30.335101 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6e0e3af-5ca4-4e3f-9b7d-6b89b07cf56f None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:30.335577 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6e0e3af-5ca4-4e3f-9b7d-6b89b07cf56f None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:30.337422 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 477/949] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:21:30 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:21:30.342586 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c83ed9ef-45e0-40c1-a5d5-3834ff239b59 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:30.343109 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c83ed9ef-45e0-40c1-a5d5-3834ff239b59 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:30.343671 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c83ed9ef-45e0-40c1-a5d5-3834ff239b59 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:30.347507 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f87c0b13-1b0e-44c0-8c46-c20da3454e0e None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:30.364247 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-c83ed9ef-45e0-40c1-a5d5-3834ff239b59 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:21:30.365850 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-c83ed9ef-45e0-40c1-a5d5-3834ff239b59 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:21:30.382873 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6355eae5-3661-48c6-ac4b-511cc40e72b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:30.383514 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6355eae5-3661-48c6-ac4b-511cc40e72b9 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:30.383794 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6355eae5-3661-48c6-ac4b-511cc40e72b9 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:30.384065 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6355eae5-3661-48c6-ac4b-511cc40e72b9 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:30.390936 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6355eae5-3661-48c6-ac4b-511cc40e72b9 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:30.392957 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6355eae5-3661-48c6-ac4b-511cc40e72b9 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:30.404407 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-6355eae5-3661-48c6-ac4b-511cc40e72b9 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 21:21:30.404407 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:30.404407 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:30.404407 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:30.404407 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:30.404407 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:30.404407 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:30.404407 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:30.404407 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:30.404407 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:30.404407 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:30.404407 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:30.404407 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 29 21:21:30.404407 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 29 21:21:30.404407 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 29 21:21:30.405321 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 29 21:21:30.405321 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:30.405321 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:30.405321 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:30.405321 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:30.405321 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:30.405321 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:30.405321 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:30.405321 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:30.405321 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:30.405321 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:30.405321 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:30.405321 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:30.405321 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:30.405321 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:30.405321 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:30.405321 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:30.411217 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 29 21:21:30.411217 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 29 21:21:30.411217 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 29 21:21:30.411217 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 29 21:21:30.411217 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 29 21:21:30.411217 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 29 21:21:30.411217 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 21:21:30.411217 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:21:30.411217 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 473/950] 10.222.0.90 () {56 vars in 1111 bytes} [Mon Aug 29 21:21:30 2022] GET /identity/v3/projects/service => generated 88 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:21:30.413443 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6355eae5-3661-48c6-ac4b-511cc40e72b9 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:30.460857 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 478/951] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:21:30 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 29 21:21:30.485291 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:30.531074 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6355eae5-3661-48c6-ac4b-511cc40e72b9 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:30.551153 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c83ed9ef-45e0-40c1-a5d5-3834ff239b59 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:30.569705 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f75a81d9-93a6-441b-939b-d300382b7aea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:30.570377 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f75a81d9-93a6-441b-939b-d300382b7aea None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:30.570657 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f75a81d9-93a6-441b-939b-d300382b7aea None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:30.570918 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f75a81d9-93a6-441b-939b-d300382b7aea None admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:30.571620 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f75a81d9-93a6-441b-939b-d300382b7aea None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:30.571927 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f75a81d9-93a6-441b-939b-d300382b7aea None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:30.573357 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f75a81d9-93a6-441b-939b-d300382b7aea None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:30.586536 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:30.588226 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 474/952] 10.222.0.90 () {56 vars in 1148 bytes} [Mon Aug 29 21:21:30 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:30.594324 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f75a81d9-93a6-441b-939b-d300382b7aea None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:30.666714 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c83ed9ef-45e0-40c1-a5d5-3834ff239b59 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:30.703177 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f75a81d9-93a6-441b-939b-d300382b7aea None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:30.704630 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae9a96a3-0aff-4864-a2c0-3eca2190615b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:30.705688 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae9a96a3-0aff-4864-a2c0-3eca2190615b None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:30.706219 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae9a96a3-0aff-4864-a2c0-3eca2190615b None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:30.706710 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae9a96a3-0aff-4864-a2c0-3eca2190615b None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:30.707504 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae9a96a3-0aff-4864-a2c0-3eca2190615b None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:30.709075 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae9a96a3-0aff-4864-a2c0-3eca2190615b None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:30.718648 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-ae9a96a3-0aff-4864-a2c0-3eca2190615b None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 29 21:21:30.718648 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:30.718648 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 29 21:21:30.718648 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 29 21:21:30.718648 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 29 21:21:30.718648 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 29 21:21:30.718648 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 29 21:21:30.718648 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 21:21:30.718648 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 29 21:21:30.718648 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:21:30.718648 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 29 21:21:30.718648 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:21:30.718648 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:30.718648 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:30.718648 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:30.718648 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:30.718648 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:30.719730 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:30.719730 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:30.719730 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:30.719730 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:30.719730 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:30.719730 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:30.719730 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 29 21:21:30.719730 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 29 21:21:30.719730 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 29 21:21:30.719730 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 29 21:21:30.719730 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:30.719730 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:30.719730 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:30.719730 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:30.719730 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:30.719730 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:30.722336 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:30.722336 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:30.722336 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:30.722336 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:30.722336 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:30.722336 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:30.722336 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:30.722336 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:30.722336 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:30.722336 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:30.722336 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 29 21:21:30.722336 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 21:21:30.722336 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 29 21:21:30.722336 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:21:30.724858 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 479/953] 10.222.0.90 () {56 vars in 1126 bytes} [Mon Aug 29 21:21:30 2022] GET /identity/v3/services/object-store => generated 93 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 29 21:21:30.726298 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:30.731779 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae9a96a3-0aff-4864-a2c0-3eca2190615b None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:30.743529 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55df7068-4ad5-4495-bad8-0422c239d4e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:30.744207 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55df7068-4ad5-4495-bad8-0422c239d4e8 None admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:30.744490 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55df7068-4ad5-4495-bad8-0422c239d4e8 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:30.744768 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55df7068-4ad5-4495-bad8-0422c239d4e8 None admin] PATH_INFO: `/v3/projects/ec03e112e3d547fbab6faf7b485c3fca/users/0c0b0da87b124de1be00faec4bbdb404/roles/9c919ca974684bc492e75274b81c78ad` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:30.747260 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55df7068-4ad5-4495-bad8-0422c239d4e8 None admin] RBAC: Authorizing `identity:create_grant(project_id=ec03e112e3d547fbab6faf7b485c3fca, user_id=0c0b0da87b124de1be00faec4bbdb404, role_id=9c919ca974684bc492e75274b81c78ad)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:30.748801 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55df7068-4ad5-4495-bad8-0422c239d4e8 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:30.763654 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 475/954] 10.222.0.90 () {58 vars in 1439 bytes} [Mon Aug 29 21:21:30 2022] PUT /identity/v3/projects/ec03e112e3d547fbab6faf7b485c3fca/users/0c0b0da87b124de1be00faec4bbdb404/roles/9c919ca974684bc492e75274b81c78ad => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:21:30.764581 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55df7068-4ad5-4495-bad8-0422c239d4e8 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:30.850766 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:30.883788 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae9a96a3-0aff-4864-a2c0-3eca2190615b None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:30.909991 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55df7068-4ad5-4495-bad8-0422c239d4e8 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:30.935221 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4794a46e-c179-4c71-944f-cb031bc93ca2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:30.936425 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4794a46e-c179-4c71-944f-cb031bc93ca2 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:30.936709 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4794a46e-c179-4c71-944f-cb031bc93ca2 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:30.936988 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4794a46e-c179-4c71-944f-cb031bc93ca2 None admin] PATH_INFO: `/v3/services` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:30.937629 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4794a46e-c179-4c71-944f-cb031bc93ca2 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:30.938005 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4794a46e-c179-4c71-944f-cb031bc93ca2 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:30.939999 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4794a46e-c179-4c71-944f-cb031bc93ca2 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:30.949837 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 480/955] 10.222.0.90 () {56 vars in 1122 bytes} [Mon Aug 29 21:21:30 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:21:30.952666 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4794a46e-c179-4c71-944f-cb031bc93ca2 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:30.956689 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b102a573-a2f3-4e96-95e9-f81086b5d579 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:30.957418 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b102a573-a2f3-4e96-95e9-f81086b5d579 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:30.957766 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b102a573-a2f3-4e96-95e9-f81086b5d579 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:30.958073 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b102a573-a2f3-4e96-95e9-f81086b5d579 None admin] PATH_INFO: `/v3/services/compute` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:30.958658 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b102a573-a2f3-4e96-95e9-f81086b5d579 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:30.960007 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b102a573-a2f3-4e96-95e9-f81086b5d579 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:30.969001 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-b102a573-a2f3-4e96-95e9-f81086b5d579 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Aug 29 21:21:30.969001 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:30.969001 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 29 21:21:30.969001 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 29 21:21:30.969001 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 29 21:21:30.969001 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 29 21:21:30.969001 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 29 21:21:30.969001 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 21:21:30.969001 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Aug 29 21:21:30.969001 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:21:30.969001 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 29 21:21:30.969001 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:21:30.969001 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:30.969001 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:30.969001 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:30.969001 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:30.969001 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:30.970999 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:30.970999 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:30.970999 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:30.970999 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:30.970999 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:30.970999 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:30.970999 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 29 21:21:30.970999 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 29 21:21:30.970999 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 29 21:21:30.970999 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 29 21:21:30.970999 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:30.970999 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:30.970999 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:30.970999 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:30.970999 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:30.970999 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:30.971904 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:30.971904 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:30.971904 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:30.971904 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:30.971904 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:30.971904 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:30.971904 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:30.971904 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:30.971904 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:30.971904 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:30.971904 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 29 21:21:30.971904 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 21:21:30.971904 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Aug 29 21:21:30.971904 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:21:30.971904 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 476/956] 10.222.0.90 () {56 vars in 1111 bytes} [Mon Aug 29 21:21:30 2022] GET /identity/v3/services/compute => generated 88 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:21:30.976524 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b102a573-a2f3-4e96-95e9-f81086b5d579 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:31.028278 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:31.096496 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4794a46e-c179-4c71-944f-cb031bc93ca2 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:31.096821 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b102a573-a2f3-4e96-95e9-f81086b5d579 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:31.134887 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4f9e722-3af5-4192-95cf-4ab47e52370d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:31.135487 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4f9e722-3af5-4192-95cf-4ab47e52370d None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:31.135772 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4f9e722-3af5-4192-95cf-4ab47e52370d None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:31.135968 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71b3a24c-c64e-4648-9b2e-95455a973db2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:31.136107 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4f9e722-3af5-4192-95cf-4ab47e52370d None admin] PATH_INFO: `/v3/services` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:31.136733 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4f9e722-3af5-4192-95cf-4ab47e52370d None admin] RBAC: Adding query filter params (name=compute) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:31.136858 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b3a24c-c64e-4648-9b2e-95455a973db2 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:31.137045 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4f9e722-3af5-4192-95cf-4ab47e52370d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:31.137272 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b3a24c-c64e-4648-9b2e-95455a973db2 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:31.137636 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b3a24c-c64e-4648-9b2e-95455a973db2 None admin] PATH_INFO: `/v3/services` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:31.138386 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71b3a24c-c64e-4648-9b2e-95455a973db2 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:31.138512 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4f9e722-3af5-4192-95cf-4ab47e52370d None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:31.138855 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71b3a24c-c64e-4648-9b2e-95455a973db2 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:31.140490 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71b3a24c-c64e-4648-9b2e-95455a973db2 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:31.144602 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 477/957] 10.222.0.90 () {56 vars in 1112 bytes} [Mon Aug 29 21:21:30 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:21:31.150321 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4f9e722-3af5-4192-95cf-4ab47e52370d None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:31.150670 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 481/958] 10.222.0.90 () {56 vars in 1122 bytes} [Mon Aug 29 21:21:30 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:31.156361 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71b3a24c-c64e-4648-9b2e-95455a973db2 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:31.174498 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:31.268897 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71b3a24c-c64e-4648-9b2e-95455a973db2 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:31.286435 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4f9e722-3af5-4192-95cf-4ab47e52370d None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:31.289251 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:31.312345 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fc2549f-16a6-4d6e-9b1c-c963017deb42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:31.314085 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fc2549f-16a6-4d6e-9b1c-c963017deb42 None admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:31.314983 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fc2549f-16a6-4d6e-9b1c-c963017deb42 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:31.315737 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fc2549f-16a6-4d6e-9b1c-c963017deb42 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:31.316638 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fc2549f-16a6-4d6e-9b1c-c963017deb42 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:31.319203 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fc2549f-16a6-4d6e-9b1c-c963017deb42 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:31.326240 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1564d4c-eaf4-4737-bbf6-681ff76ce395 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:31.326944 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1564d4c-eaf4-4737-bbf6-681ff76ce395 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:31.327341 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1564d4c-eaf4-4737-bbf6-681ff76ce395 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:31.327699 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1564d4c-eaf4-4737-bbf6-681ff76ce395 None admin] PATH_INFO: `/v3/services` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:31.328451 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1564d4c-eaf4-4737-bbf6-681ff76ce395 None admin] RBAC: Adding query filter params (type=compute) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:31.328814 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1564d4c-eaf4-4737-bbf6-681ff76ce395 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:31.331038 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1564d4c-eaf4-4737-bbf6-681ff76ce395 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:31.343046 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 478/959] 10.222.0.90 () {56 vars in 1112 bytes} [Mon Aug 29 21:21:31 2022] GET /identity/v3/services?type=compute => generated 124 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:21:31.346083 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 482/960] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:21:31 2022] POST /identity/v3/endpoints => generated 349 bytes in 190 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 21:21:31.683622 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1c43456-64e9-477b-997b-34f4bf8ec60c None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:31.684025 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1c43456-64e9-477b-997b-34f4bf8ec60c None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:31.684203 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1c43456-64e9-477b-997b-34f4bf8ec60c None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:31.685828 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 479/961] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:21:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:21:31.691697 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-884fe6f2-7cf4-4d2d-81e1-1e8d408820c8 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:31.691985 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-884fe6f2-7cf4-4d2d-81e1-1e8d408820c8 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:31.692260 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-884fe6f2-7cf4-4d2d-81e1-1e8d408820c8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:31.711557 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-884fe6f2-7cf4-4d2d-81e1-1e8d408820c8 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:21:31.712747 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-884fe6f2-7cf4-4d2d-81e1-1e8d408820c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:21:31.779846 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:31.793055 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 483/962] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:21:31 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 21:21:32.071049 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1c43456-64e9-477b-997b-34f4bf8ec60c None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:32.175699 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1c43456-64e9-477b-997b-34f4bf8ec60c None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:32.209759 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48787546-d727-4690-a0ab-15ae9c6d6de0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:32.210388 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48787546-d727-4690-a0ab-15ae9c6d6de0 None admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:32.210663 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48787546-d727-4690-a0ab-15ae9c6d6de0 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:32.210933 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48787546-d727-4690-a0ab-15ae9c6d6de0 None admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:32.211581 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48787546-d727-4690-a0ab-15ae9c6d6de0 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:32.213002 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48787546-d727-4690-a0ab-15ae9c6d6de0 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:32.228786 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.sql.core [None req-48787546-d727-4690-a0ab-15ae9c6d6de0 None admin] Conflict user: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Aug 29 21:21:32.228786 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Aug 29 21:21:32.228786 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [parameters: {'user_id': 'e8a8303d30ed4b21adfc865e9394383c', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Aug 29 21:21:32.228786 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81434) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Aug 29 21:21:32.236724 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-48787546-d727-4690-a0ab-15ae9c6d6de0 None admin] Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default.: keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Aug 29 21:21:32.236724 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:32.236724 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Aug 29 21:21:32.236724 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application self.dialect.do_execute( Aug 29 21:21:32.236724 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Aug 29 21:21:32.236724 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Aug 29 21:21:32.236724 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Aug 29 21:21:32.236724 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application result = self._query(query) Aug 29 21:21:32.236724 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Aug 29 21:21:32.236724 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application conn.query(q) Aug 29 21:21:32.236724 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Aug 29 21:21:32.236724 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Aug 29 21:21:32.236724 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Aug 29 21:21:32.236724 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application result.read() Aug 29 21:21:32.236724 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Aug 29 21:21:32.237649 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Aug 29 21:21:32.237649 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Aug 29 21:21:32.237649 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application packet.raise_for_error() Aug 29 21:21:32.237649 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Aug 29 21:21:32.237649 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Aug 29 21:21:32.237649 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Aug 29 21:21:32.237649 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Aug 29 21:21:32.237649 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Aug 29 21:21:32.237649 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:21:32.237649 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Aug 29 21:21:32.237649 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:21:32.237649 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:32.237649 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Aug 29 21:21:32.237649 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return method(*args, **kwargs) Aug 29 21:21:32.237649 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 141, in create_user Aug 29 21:21:32.237649 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return base.filter_user(user_ref.to_dict()) Aug 29 21:21:32.237649 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Aug 29 21:21:32.237649 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application next(self.gen) Aug 29 21:21:32.238714 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Aug 29 21:21:32.238714 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application yield resource Aug 29 21:21:32.238714 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Aug 29 21:21:32.238714 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application next(self.gen) Aug 29 21:21:32.238714 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Aug 29 21:21:32.238714 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application self.session.rollback() Aug 29 21:21:32.238714 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Aug 29 21:21:32.238714 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application self.force_reraise() Aug 29 21:21:32.238714 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Aug 29 21:21:32.238714 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise self.value Aug 29 21:21:32.238714 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Aug 29 21:21:32.238714 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Aug 29 21:21:32.238714 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Aug 29 21:21:32.238714 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application session.commit() Aug 29 21:21:32.238714 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Aug 29 21:21:32.238714 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Aug 29 21:21:32.238714 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Aug 29 21:21:32.239637 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application self._prepare_impl() Aug 29 21:21:32.239637 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Aug 29 21:21:32.239637 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application self.session.flush() Aug 29 21:21:32.239637 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Aug 29 21:21:32.239637 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application self._flush(objects) Aug 29 21:21:32.239637 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Aug 29 21:21:32.239637 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Aug 29 21:21:32.239637 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Aug 29 21:21:32.239637 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application compat.raise_( Aug 29 21:21:32.239637 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Aug 29 21:21:32.239637 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception Aug 29 21:21:32.239637 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Aug 29 21:21:32.239637 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application flush_context.execute() Aug 29 21:21:32.239637 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Aug 29 21:21:32.239637 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rec.execute(self) Aug 29 21:21:32.239637 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Aug 29 21:21:32.239637 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Aug 29 21:21:32.240574 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Aug 29 21:21:32.240574 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application _emit_insert_statements( Aug 29 21:21:32.240574 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1238, in _emit_insert_statements Aug 29 21:21:32.240574 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application result = connection._execute_20( Aug 29 21:21:32.240574 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Aug 29 21:21:32.240574 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Aug 29 21:21:32.240574 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Aug 29 21:21:32.240574 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Aug 29 21:21:32.240574 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Aug 29 21:21:32.240574 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ret = self._execute_context( Aug 29 21:21:32.240574 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Aug 29 21:21:32.240574 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Aug 29 21:21:32.240574 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Aug 29 21:21:32.240574 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Aug 29 21:21:32.240574 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Aug 29 21:21:32.240574 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception Aug 29 21:21:32.241458 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Aug 29 21:21:32.241458 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application self.dialect.do_execute( Aug 29 21:21:32.241458 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Aug 29 21:21:32.241458 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Aug 29 21:21:32.241458 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Aug 29 21:21:32.241458 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application result = self._query(query) Aug 29 21:21:32.241458 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Aug 29 21:21:32.241458 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application conn.query(q) Aug 29 21:21:32.241458 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Aug 29 21:21:32.241458 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Aug 29 21:21:32.241458 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Aug 29 21:21:32.241458 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application result.read() Aug 29 21:21:32.241458 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Aug 29 21:21:32.241458 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Aug 29 21:21:32.241458 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Aug 29 21:21:32.241458 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application packet.raise_for_error() Aug 29 21:21:32.243661 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Aug 29 21:21:32.243661 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Aug 29 21:21:32.243661 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Aug 29 21:21:32.243661 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Aug 29 21:21:32.243661 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Aug 29 21:21:32.243661 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Aug 29 21:21:32.243661 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [parameters: {'user_id': 'e8a8303d30ed4b21adfc865e9394383c', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Aug 29 21:21:32.243661 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Aug 29 21:21:32.243661 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:21:32.243661 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 29 21:21:32.243661 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:21:32.243661 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:32.243661 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:32.243661 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:32.243661 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:32.244726 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:32.244726 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:32.244726 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:32.244726 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:32.244726 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:32.244726 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:32.244726 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:32.244726 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 244, in post Aug 29 21:21:32.244726 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.create_user( Aug 29 21:21:32.244726 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:32.244726 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:32.244726 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 29 21:21:32.244726 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 21:21:32.244726 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 29 21:21:32.244726 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 21:21:32.244726 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1000, in create_user Aug 29 21:21:32.244726 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ref = self._create_user_with_federated_objects(user, driver) Aug 29 21:21:32.245908 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 966, in _create_user_with_federated_objects Aug 29 21:21:32.245908 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application user = driver.create_user(user['id'], user) Aug 29 21:21:32.245908 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Aug 29 21:21:32.245908 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Aug 29 21:21:32.245908 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Aug 29 21:21:32.245908 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:21:32.245908 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 480/963] 10.222.0.90 () {60 vars in 1131 bytes} [Mon Aug 29 21:21:32 2022] POST /identity/v3/users => generated 160 bytes in 168 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:21:32.246529 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-884fe6f2-7cf4-4d2d-81e1-1e8d408820c8 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:32.360105 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-884fe6f2-7cf4-4d2d-81e1-1e8d408820c8 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:32.398625 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-334d7926-1797-4ba5-84c8-c50fa139294d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:32.399273 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-334d7926-1797-4ba5-84c8-c50fa139294d None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:32.399555 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-334d7926-1797-4ba5-84c8-c50fa139294d None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:32.399832 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-334d7926-1797-4ba5-84c8-c50fa139294d None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:32.409882 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-334d7926-1797-4ba5-84c8-c50fa139294d None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:32.411674 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-334d7926-1797-4ba5-84c8-c50fa139294d None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:32.424204 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-334d7926-1797-4ba5-84c8-c50fa139294d None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Aug 29 21:21:32.424204 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:32.424204 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:32.424204 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:32.424204 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:32.424204 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:32.424204 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:32.424204 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:32.424204 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:32.424204 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:32.424204 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:32.424204 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:32.424204 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 29 21:21:32.424204 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 29 21:21:32.424204 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 29 21:21:32.424204 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 29 21:21:32.425199 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:32.425199 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:32.425199 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 29 21:21:32.425199 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 21:21:32.425199 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 29 21:21:32.425199 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 21:21:32.425199 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:32.425199 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:32.425199 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:32.425199 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:32.425199 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:32.425199 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:32.425199 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:32.425199 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:32.425199 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:32.425199 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:32.425199 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:32.425199 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:32.426210 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:32.426210 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:32.426210 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 29 21:21:32.426210 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 29 21:21:32.426210 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 29 21:21:32.426210 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 29 21:21:32.426210 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 29 21:21:32.426210 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 29 21:21:32.426210 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Aug 29 21:21:32.426210 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:21:32.428846 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 484/964] 10.222.0.90 () {56 vars in 1099 bytes} [Mon Aug 29 21:21:32 2022] GET /identity/v3/users/cinder => generated 84 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:21:32.435073 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48787546-d727-4690-a0ab-15ae9c6d6de0 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:32.471897 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:32.552661 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48787546-d727-4690-a0ab-15ae9c6d6de0 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:32.587699 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34ee3c13-e0da-4476-b936-b4e99ceb55a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:32.588390 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34ee3c13-e0da-4476-b936-b4e99ceb55a7 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:32.588692 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34ee3c13-e0da-4476-b936-b4e99ceb55a7 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:32.588952 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34ee3c13-e0da-4476-b936-b4e99ceb55a7 None admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:32.589673 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34ee3c13-e0da-4476-b936-b4e99ceb55a7 None admin] RBAC: Adding query filter params (name=cinder) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:32.590005 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34ee3c13-e0da-4476-b936-b4e99ceb55a7 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:32.591788 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34ee3c13-e0da-4476-b936-b4e99ceb55a7 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:32.632406 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 481/965] 10.222.0.90 () {56 vars in 1101 bytes} [Mon Aug 29 21:21:32 2022] GET /identity/v3/users?name=cinder => generated 354 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:32.785060 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df4ac688-7d71-463f-a437-019219aef2d1 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:32.785417 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df4ac688-7d71-463f-a437-019219aef2d1 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:32.785600 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df4ac688-7d71-463f-a437-019219aef2d1 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:32.787343 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 485/966] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:21:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:21:32.792601 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44f22b32-8c51-490e-bb89-8457f65acfc1 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:32.792896 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44f22b32-8c51-490e-bb89-8457f65acfc1 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:32.793172 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44f22b32-8c51-490e-bb89-8457f65acfc1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:32.813191 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-44f22b32-8c51-490e-bb89-8457f65acfc1 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:21:32.815273 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-44f22b32-8c51-490e-bb89-8457f65acfc1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:21:32.894269 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 482/967] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:21:32 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 21:21:33.168943 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df4ac688-7d71-463f-a437-019219aef2d1 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:33.230475 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:33.278846 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df4ac688-7d71-463f-a437-019219aef2d1 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:33.313995 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8bf6cd6-96d4-46c7-b057-09a6cf9fcebb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:33.314647 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8bf6cd6-96d4-46c7-b057-09a6cf9fcebb None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:33.314929 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8bf6cd6-96d4-46c7-b057-09a6cf9fcebb None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:33.315204 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8bf6cd6-96d4-46c7-b057-09a6cf9fcebb None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:33.326227 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8bf6cd6-96d4-46c7-b057-09a6cf9fcebb None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 21:21:33.326767 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed8e17d-e85d-445b-ba58-ce950a7c69e8 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:33.327208 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed8e17d-e85d-445b-ba58-ce950a7c69e8 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:33.327331 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed8e17d-e85d-445b-ba58-ce950a7c69e8 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:33.328777 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 483/968] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:21:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:21:33.331205 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-e8bf6cd6-96d4-46c7-b057-09a6cf9fcebb None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 29 21:21:33.331205 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:33.331205 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:33.331205 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:33.331205 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:33.331205 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:33.331205 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:33.331205 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:33.331205 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:33.331205 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:33.331205 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:33.331205 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:33.331205 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 21:21:33.331205 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 21:21:33.331205 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 21:21:33.331205 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 21:21:33.332193 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 21:21:33.332193 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 21:21:33.332193 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 21:21:33.332193 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 21:21:33.332193 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:33.332193 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:33.332193 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:33.332193 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:33.332193 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:33.332193 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:33.332193 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:33.332193 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:33.332193 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:33.332193 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:33.332193 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:33.332193 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:33.333093 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:33.333093 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:33.333093 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:33.333093 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:33.333093 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 21:21:33.333093 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 21:21:33.333093 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 21:21:33.333093 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 21:21:33.333093 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 21:21:33.333093 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 21:21:33.333093 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 29 21:21:33.333093 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:21:33.333093 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 486/969] 10.222.0.90 () {56 vars in 1120 bytes} [Mon Aug 29 21:21:33 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:21:33.333870 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59fc5809-ea6b-432b-9405-c07eb8fdc783 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:33.334450 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59fc5809-ea6b-432b-9405-c07eb8fdc783 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:33.334450 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59fc5809-ea6b-432b-9405-c07eb8fdc783 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:33.338197 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8bf6cd6-96d4-46c7-b057-09a6cf9fcebb None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:33.353843 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-59fc5809-ea6b-432b-9405-c07eb8fdc783 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:21:33.355151 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-59fc5809-ea6b-432b-9405-c07eb8fdc783 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:21:33.370049 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:33.443333 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 484/970] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:21:33 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 21:21:33.458136 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41bdac2b-6ebf-45e5-b33d-cc9951b07835 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:33.458454 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41bdac2b-6ebf-45e5-b33d-cc9951b07835 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:33.458960 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41bdac2b-6ebf-45e5-b33d-cc9951b07835 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:33.461277 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 485/971] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:21:33 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 29 21:21:33.462422 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8bf6cd6-96d4-46c7-b057-09a6cf9fcebb None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:33.466619 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa09597e-f977-492b-b1da-34f083c6484e None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:33.467126 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa09597e-f977-492b-b1da-34f083c6484e None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:33.467632 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa09597e-f977-492b-b1da-34f083c6484e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:33.480073 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:33.489321 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-aa09597e-f977-492b-b1da-34f083c6484e None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:21:33.490561 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-aa09597e-f977-492b-b1da-34f083c6484e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:21:33.503696 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-472069da-e618-4431-b4a3-6833762bd216 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:33.504315 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-472069da-e618-4431-b4a3-6833762bd216 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:33.504596 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-472069da-e618-4431-b4a3-6833762bd216 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:33.504983 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-472069da-e618-4431-b4a3-6833762bd216 None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:33.505776 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-472069da-e618-4431-b4a3-6833762bd216 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:33.506110 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-472069da-e618-4431-b4a3-6833762bd216 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:33.507540 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-472069da-e618-4431-b4a3-6833762bd216 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:33.521302 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 487/972] 10.222.0.90 () {56 vars in 1115 bytes} [Mon Aug 29 21:21:33 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:21:33.522299 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-472069da-e618-4431-b4a3-6833762bd216 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:33.576449 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 486/973] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:21:33 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 21:21:33.626873 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-472069da-e618-4431-b4a3-6833762bd216 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:33.661144 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff67604b-2f90-416d-bc42-7caaf14b14ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:33.661768 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff67604b-2f90-416d-bc42-7caaf14b14ad None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:33.662087 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff67604b-2f90-416d-bc42-7caaf14b14ad None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:33.662364 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff67604b-2f90-416d-bc42-7caaf14b14ad None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:33.673436 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff67604b-2f90-416d-bc42-7caaf14b14ad None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:33.674904 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff67604b-2f90-416d-bc42-7caaf14b14ad None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:33.690386 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-ff67604b-2f90-416d-bc42-7caaf14b14ad None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:21:33.690386 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:33.690386 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:33.690386 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:33.690386 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:33.690386 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:33.690386 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:33.690386 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:33.690386 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:33.690386 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:33.690386 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:33.690386 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:33.690386 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 21:21:33.690386 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 21:21:33.690386 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 21:21:33.690386 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 21:21:33.691399 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:33.691399 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:33.691399 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:33.691399 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:33.691399 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:33.691399 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:33.691399 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:33.691399 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:33.691399 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:33.691399 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:33.691399 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:33.691399 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:33.691399 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:33.691399 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:33.691399 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:33.691399 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:33.691399 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 21:21:33.697649 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 21:21:33.697649 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:21:33.697649 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:21:33.697649 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 488/974] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:21:33 2022] GET /identity/v3/domains/Default => generated 87 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:21:33.697947 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa09597e-f977-492b-b1da-34f083c6484e None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:33.710687 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff67604b-2f90-416d-bc42-7caaf14b14ad None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:33.744482 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:33.788898 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:33.816242 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa09597e-f977-492b-b1da-34f083c6484e None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:33.826257 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff67604b-2f90-416d-bc42-7caaf14b14ad None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:33.855784 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c744aa65-67f4-45ff-b095-4bf5d6dd00f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:33.856733 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c744aa65-67f4-45ff-b095-4bf5d6dd00f6 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:33.857025 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c744aa65-67f4-45ff-b095-4bf5d6dd00f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:33.857304 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c744aa65-67f4-45ff-b095-4bf5d6dd00f6 None admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:33.857997 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c744aa65-67f4-45ff-b095-4bf5d6dd00f6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:33.858305 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c744aa65-67f4-45ff-b095-4bf5d6dd00f6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:33.859679 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c744aa65-67f4-45ff-b095-4bf5d6dd00f6 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:33.865566 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7bdac18-04c1-4bc1-b91a-9f4a54ec51be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:33.866221 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7bdac18-04c1-4bc1-b91a-9f4a54ec51be None admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:33.866664 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7bdac18-04c1-4bc1-b91a-9f4a54ec51be None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:33.866991 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7bdac18-04c1-4bc1-b91a-9f4a54ec51be None admin] PATH_INFO: `/v3/services` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:33.867433 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7bdac18-04c1-4bc1-b91a-9f4a54ec51be None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:33.868780 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7bdac18-04c1-4bc1-b91a-9f4a54ec51be None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:33.872353 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 487/975] 10.222.0.90 () {56 vars in 1109 bytes} [Mon Aug 29 21:21:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:33.872973 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c744aa65-67f4-45ff-b095-4bf5d6dd00f6 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:33.893949 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 489/976] 10.222.0.90 () {60 vars in 1141 bytes} [Mon Aug 29 21:21:33 2022] POST /identity/v3/services => generated 242 bytes in 184 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 21:21:33.894903 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7bdac18-04c1-4bc1-b91a-9f4a54ec51be None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:34.004096 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c744aa65-67f4-45ff-b095-4bf5d6dd00f6 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:34.006401 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:34.027382 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7bdac18-04c1-4bc1-b91a-9f4a54ec51be None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:34.045189 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e875a5e2-897f-407a-8e6d-09ef50240b29 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:34.046159 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e875a5e2-897f-407a-8e6d-09ef50240b29 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:34.046439 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e875a5e2-897f-407a-8e6d-09ef50240b29 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:34.047623 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e875a5e2-897f-407a-8e6d-09ef50240b29 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:34.047623 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e875a5e2-897f-407a-8e6d-09ef50240b29 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:34.048932 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e875a5e2-897f-407a-8e6d-09ef50240b29 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:34.058142 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-e875a5e2-897f-407a-8e6d-09ef50240b29 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 29 21:21:34.058142 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:34.058142 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 29 21:21:34.058142 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 29 21:21:34.058142 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 29 21:21:34.058142 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 29 21:21:34.058142 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 29 21:21:34.058142 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 21:21:34.058142 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 29 21:21:34.058142 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:21:34.058142 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 29 21:21:34.058142 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:21:34.058142 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:34.058142 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:34.058142 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:34.058142 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:34.058142 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:34.059168 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:34.059168 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:34.059168 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:34.059168 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:34.059168 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:34.059168 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:34.059168 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 29 21:21:34.059168 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 29 21:21:34.059168 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 29 21:21:34.059168 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 29 21:21:34.059168 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:34.059168 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:34.059168 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:34.059168 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:34.059168 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:34.059168 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:34.060172 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:34.060172 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:34.060172 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:34.060172 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:34.060172 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:34.060172 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:34.060172 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:34.060172 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:34.060172 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:34.060172 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:34.060172 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 29 21:21:34.060172 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 21:21:34.060172 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 29 21:21:34.060172 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:21:34.060172 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 488/977] 10.222.0.90 () {56 vars in 1126 bytes} [Mon Aug 29 21:21:33 2022] GET /identity/v3/services/object-store => generated 93 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:21:34.066371 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e875a5e2-897f-407a-8e6d-09ef50240b29 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:34.069645 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:34.070484 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ad45a7d-1b31-4b4b-bf34-9b9b8489cf92 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:34.070957 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ad45a7d-1b31-4b4b-bf34-9b9b8489cf92 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:34.071222 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ad45a7d-1b31-4b4b-bf34-9b9b8489cf92 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:34.071652 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ad45a7d-1b31-4b4b-bf34-9b9b8489cf92 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:34.081005 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ad45a7d-1b31-4b4b-bf34-9b9b8489cf92 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:34.082496 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ad45a7d-1b31-4b4b-bf34-9b9b8489cf92 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:34.093972 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-4ad45a7d-1b31-4b4b-bf34-9b9b8489cf92 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 29 21:21:34.093972 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:34.093972 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:34.093972 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:34.093972 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:34.093972 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:34.093972 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:34.093972 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:34.093972 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:34.093972 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:34.093972 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:34.093972 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:34.093972 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 29 21:21:34.093972 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 29 21:21:34.093972 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 29 21:21:34.093972 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 29 21:21:34.095033 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:34.095033 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:34.095033 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 29 21:21:34.095033 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 21:21:34.095033 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 29 21:21:34.095033 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 21:21:34.095033 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:34.095033 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:34.095033 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:34.095033 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:34.095033 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:34.095033 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:34.095033 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:34.095033 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:34.095033 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:34.095033 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:34.095033 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:34.095033 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:34.097288 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:34.097288 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:34.097288 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 29 21:21:34.097288 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 29 21:21:34.097288 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 29 21:21:34.097288 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 29 21:21:34.097288 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 29 21:21:34.097288 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 29 21:21:34.097288 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 29 21:21:34.097288 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:21:34.097288 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 490/978] 10.222.0.90 () {56 vars in 1117 bytes} [Mon Aug 29 21:21:33 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:21:34.101110 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ad45a7d-1b31-4b4b-bf34-9b9b8489cf92 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:34.178950 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e875a5e2-897f-407a-8e6d-09ef50240b29 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:34.216041 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8f4d681-4483-430d-8479-8196d5694e22 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:34.216664 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f4d681-4483-430d-8479-8196d5694e22 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:34.216789 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ad45a7d-1b31-4b4b-bf34-9b9b8489cf92 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:34.216970 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f4d681-4483-430d-8479-8196d5694e22 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:34.217209 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f4d681-4483-430d-8479-8196d5694e22 None admin] PATH_INFO: `/v3/services` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:34.217878 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8f4d681-4483-430d-8479-8196d5694e22 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:34.218356 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8f4d681-4483-430d-8479-8196d5694e22 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:34.221009 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8f4d681-4483-430d-8479-8196d5694e22 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:34.229960 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 489/979] 10.222.0.90 () {56 vars in 1122 bytes} [Mon Aug 29 21:21:34 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:21:34.233640 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8f4d681-4483-430d-8479-8196d5694e22 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:34.260397 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06cda7b9-a303-4812-8e82-6cf0503f412e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:34.261016 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06cda7b9-a303-4812-8e82-6cf0503f412e None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:34.261299 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06cda7b9-a303-4812-8e82-6cf0503f412e None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:34.261568 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06cda7b9-a303-4812-8e82-6cf0503f412e None admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:34.262328 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06cda7b9-a303-4812-8e82-6cf0503f412e None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:34.262675 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06cda7b9-a303-4812-8e82-6cf0503f412e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:34.264172 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06cda7b9-a303-4812-8e82-6cf0503f412e None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:34.290814 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 491/980] 10.222.0.90 () {56 vars in 1149 bytes} [Mon Aug 29 21:21:34 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:21:34.291724 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06cda7b9-a303-4812-8e82-6cf0503f412e None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:34.296691 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:34.306218 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:34.343430 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8f4d681-4483-430d-8479-8196d5694e22 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:34.379046 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ed3a758-2e36-428c-af78-c59eadad6e36 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:34.379772 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ed3a758-2e36-428c-af78-c59eadad6e36 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:34.380055 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ed3a758-2e36-428c-af78-c59eadad6e36 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:34.380331 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ed3a758-2e36-428c-af78-c59eadad6e36 None admin] PATH_INFO: `/v3/services` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:34.380973 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ed3a758-2e36-428c-af78-c59eadad6e36 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:34.381280 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ed3a758-2e36-428c-af78-c59eadad6e36 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:34.382677 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ed3a758-2e36-428c-af78-c59eadad6e36 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:34.387992 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 490/981] 10.222.0.90 () {56 vars in 1122 bytes} [Mon Aug 29 21:21:34 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:21:34.394359 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ed3a758-2e36-428c-af78-c59eadad6e36 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:34.406188 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06cda7b9-a303-4812-8e82-6cf0503f412e None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:34.445119 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82b161df-ab95-4383-b0f5-ebb9a1f3e1d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:34.445671 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82b161df-ab95-4383-b0f5-ebb9a1f3e1d8 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:34.445976 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82b161df-ab95-4383-b0f5-ebb9a1f3e1d8 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:34.446245 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82b161df-ab95-4383-b0f5-ebb9a1f3e1d8 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:34.502380 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ed3a758-2e36-428c-af78-c59eadad6e36 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:34.522032 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82b161df-ab95-4383-b0f5-ebb9a1f3e1d8 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:34.524230 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82b161df-ab95-4383-b0f5-ebb9a1f3e1d8 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:34.528353 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:34.533642 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:34.542259 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-490450cd-b4e9-4850-a516-5a912708bc43 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:34.543070 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-490450cd-b4e9-4850-a516-5a912708bc43 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:34.543369 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-490450cd-b4e9-4850-a516-5a912708bc43 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:34.543654 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-490450cd-b4e9-4850-a516-5a912708bc43 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:34.544680 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-490450cd-b4e9-4850-a516-5a912708bc43 None admin] RBAC: Adding query filter params (interface=admin, service_id=c69fc69e24984d4191af3077686e5ca3, region_id=RegionOne) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:34.544996 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-490450cd-b4e9-4850-a516-5a912708bc43 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:34.546420 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-490450cd-b4e9-4850-a516-5a912708bc43 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:34.558826 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 491/982] 10.222.0.90 () {56 vars in 1249 bytes} [Mon Aug 29 21:21:34 2022] GET /identity/v3/endpoints?service_id=c69fc69e24984d4191af3077686e5ca3&interface=admin®ion_id=RegionOne => generated 193 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:21:34.559723 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-490450cd-b4e9-4850-a516-5a912708bc43 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:34.637397 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 492/983] 10.222.0.90 () {58 vars in 1282 bytes} [Mon Aug 29 21:21:34 2022] GET /identity/v3/auth/tokens => generated 1570 bytes in 345 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 29 21:21:34.644120 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82b161df-ab95-4383-b0f5-ebb9a1f3e1d8 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:34.676690 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-490450cd-b4e9-4850-a516-5a912708bc43 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:34.712238 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44b46775-7e16-4256-a9c2-d446782d624a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:34.712855 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44b46775-7e16-4256-a9c2-d446782d624a None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:34.713139 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44b46775-7e16-4256-a9c2-d446782d624a None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:34.713414 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44b46775-7e16-4256-a9c2-d446782d624a None admin] PATH_INFO: `/v3/services` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:34.714057 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44b46775-7e16-4256-a9c2-d446782d624a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:34.715423 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44b46775-7e16-4256-a9c2-d446782d624a None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:34.722132 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 492/984] 10.222.0.90 () {56 vars in 1087 bytes} [Mon Aug 29 21:21:34 2022] GET /identity/v3/services => generated 1727 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:21:34.762311 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82b161df-ab95-4383-b0f5-ebb9a1f3e1d8 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:34.765771 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:34.803030 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87b3217e-b7a0-4634-9e11-5133dc2bfd8b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:34.803689 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87b3217e-b7a0-4634-9e11-5133dc2bfd8b None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:34.803971 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87b3217e-b7a0-4634-9e11-5133dc2bfd8b None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:34.804258 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87b3217e-b7a0-4634-9e11-5133dc2bfd8b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:34.816464 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87b3217e-b7a0-4634-9e11-5133dc2bfd8b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:34.817988 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87b3217e-b7a0-4634-9e11-5133dc2bfd8b None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:34.834678 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-87b3217e-b7a0-4634-9e11-5133dc2bfd8b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:21:34.834678 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:34.834678 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:34.834678 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:34.834678 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:34.834678 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:34.834678 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:34.834678 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:34.834678 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:34.834678 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:34.834678 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:34.834678 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:34.834678 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 21:21:34.834678 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 21:21:34.834678 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 21:21:34.834678 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 21:21:34.835716 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:34.835716 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:34.835716 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:34.835716 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:34.835716 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:34.835716 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:34.835716 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:34.835716 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:34.835716 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:34.835716 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:34.835716 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:34.835716 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:34.835716 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:34.835716 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:34.835716 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:34.835716 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:34.835716 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 21:21:34.836642 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 21:21:34.836642 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:21:34.836642 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:21:34.838117 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 493/985] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:21:34 2022] GET /identity/v3/domains/Default => generated 87 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 29 21:21:34.841720 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44b46775-7e16-4256-a9c2-d446782d624a None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:34.908306 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:34.954792 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44b46775-7e16-4256-a9c2-d446782d624a None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:34.990340 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00826fbd-3a20-4256-a8fc-c1af5f002ad1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:34.990953 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00826fbd-3a20-4256-a8fc-c1af5f002ad1 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:34.991230 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00826fbd-3a20-4256-a8fc-c1af5f002ad1 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:34.991500 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00826fbd-3a20-4256-a8fc-c1af5f002ad1 None admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:34.992144 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00826fbd-3a20-4256-a8fc-c1af5f002ad1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:34.992444 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00826fbd-3a20-4256-a8fc-c1af5f002ad1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:34.993815 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00826fbd-3a20-4256-a8fc-c1af5f002ad1 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:35.009881 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 493/986] 10.222.0.90 () {56 vars in 1109 bytes} [Mon Aug 29 21:21:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:35.011914 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87b3217e-b7a0-4634-9e11-5133dc2bfd8b None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:35.118915 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87b3217e-b7a0-4634-9e11-5133dc2bfd8b None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:35.153668 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58cdb12a-94ba-4f0c-9648-114156a1de5a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:35.154344 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58cdb12a-94ba-4f0c-9648-114156a1de5a None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:35.154624 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58cdb12a-94ba-4f0c-9648-114156a1de5a None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:35.154896 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58cdb12a-94ba-4f0c-9648-114156a1de5a None admin] PATH_INFO: `/v3/projects/service` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:35.162280 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58cdb12a-94ba-4f0c-9648-114156a1de5a None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:35.163732 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58cdb12a-94ba-4f0c-9648-114156a1de5a None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:35.176114 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-58cdb12a-94ba-4f0c-9648-114156a1de5a None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 21:21:35.176114 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:35.176114 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:35.176114 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:35.176114 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:35.176114 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:35.176114 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:35.176114 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:35.176114 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:35.176114 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:35.176114 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:35.176114 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:35.176114 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 29 21:21:35.176114 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 29 21:21:35.176114 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 29 21:21:35.177033 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 29 21:21:35.177033 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:35.177033 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:35.177033 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:35.177033 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:35.177033 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:35.177033 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:35.177033 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:35.177033 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:35.177033 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:35.177033 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:35.177033 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:35.177033 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:35.177033 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:35.177033 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:35.177033 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:35.177033 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:35.177985 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 29 21:21:35.177985 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 29 21:21:35.177985 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 29 21:21:35.177985 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 29 21:21:35.177985 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 29 21:21:35.177985 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 29 21:21:35.177985 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 21:21:35.177985 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:21:35.177985 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 494/987] 10.222.0.90 () {56 vars in 1111 bytes} [Mon Aug 29 21:21:35 2022] GET /identity/v3/projects/service => generated 88 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 29 21:21:35.183239 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00826fbd-3a20-4256-a8fc-c1af5f002ad1 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:35.290547 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00826fbd-3a20-4256-a8fc-c1af5f002ad1 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:35.308983 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:35.327000 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-864cf4e7-0235-4725-b4f7-09e17f05f996 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:35.327607 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-864cf4e7-0235-4725-b4f7-09e17f05f996 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:35.327883 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-864cf4e7-0235-4725-b4f7-09e17f05f996 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:35.328152 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-864cf4e7-0235-4725-b4f7-09e17f05f996 None admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:35.328840 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-864cf4e7-0235-4725-b4f7-09e17f05f996 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:35.329285 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-864cf4e7-0235-4725-b4f7-09e17f05f996 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:35.330851 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-864cf4e7-0235-4725-b4f7-09e17f05f996 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:35.343483 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 494/988] 10.222.0.90 () {56 vars in 1148 bytes} [Mon Aug 29 21:21:35 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:35.349645 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58cdb12a-94ba-4f0c-9648-114156a1de5a None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:35.369608 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:35.465287 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58cdb12a-94ba-4f0c-9648-114156a1de5a None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:35.499528 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc27569f-ab44-4c5e-b66d-e3c48e542a4b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:35.500139 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc27569f-ab44-4c5e-b66d-e3c48e542a4b None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:35.500423 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc27569f-ab44-4c5e-b66d-e3c48e542a4b None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:35.500699 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc27569f-ab44-4c5e-b66d-e3c48e542a4b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:35.502413 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc27569f-ab44-4c5e-b66d-e3c48e542a4b None admin] RBAC: Adding query filter params (role.id=9c919ca974684bc492e75274b81c78ad, scope.project.id=ec03e112e3d547fbab6faf7b485c3fca, user.id=0c0b0da87b124de1be00faec4bbdb404) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:35.502709 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc27569f-ab44-4c5e-b66d-e3c48e542a4b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:35.504608 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc27569f-ab44-4c5e-b66d-e3c48e542a4b None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:35.510831 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 495/989] 10.222.0.90 () {56 vars in 1374 bytes} [Mon Aug 29 21:21:35 2022] GET /identity/v3/role_assignments?user.id=0c0b0da87b124de1be00faec4bbdb404&scope.project.id=ec03e112e3d547fbab6faf7b485c3fca&role.id=9c919ca974684bc492e75274b81c78ad => generated 608 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:21:35.908916 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a07e4f8-a0a7-4ff5-8483-4e9969a4ca87 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:35.909243 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a07e4f8-a0a7-4ff5-8483-4e9969a4ca87 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:35.909481 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a07e4f8-a0a7-4ff5-8483-4e9969a4ca87 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:35.911004 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 495/990] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:21:35 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:21:35.915897 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f56eb66d-57b5-4e20-94ce-8efab1023f3d None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:35.916189 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f56eb66d-57b5-4e20-94ce-8efab1023f3d None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:35.916466 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f56eb66d-57b5-4e20-94ce-8efab1023f3d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:35.935782 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-f56eb66d-57b5-4e20-94ce-8efab1023f3d None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:21:35.936984 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-f56eb66d-57b5-4e20-94ce-8efab1023f3d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:21:36.018790 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 496/991] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:21:35 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 21:21:36.287408 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a07e4f8-a0a7-4ff5-8483-4e9969a4ca87 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:36.396612 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a07e4f8-a0a7-4ff5-8483-4e9969a4ca87 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:36.431309 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6271090-b9cb-411d-97ac-0cccd082d826 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:36.431953 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6271090-b9cb-411d-97ac-0cccd082d826 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:36.432239 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6271090-b9cb-411d-97ac-0cccd082d826 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:36.432516 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6271090-b9cb-411d-97ac-0cccd082d826 None admin] PATH_INFO: `/v3/services/compute` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:36.433108 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6271090-b9cb-411d-97ac-0cccd082d826 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:36.434500 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6271090-b9cb-411d-97ac-0cccd082d826 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:36.444612 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-c6271090-b9cb-411d-97ac-0cccd082d826 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Aug 29 21:21:36.444612 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:36.444612 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 29 21:21:36.444612 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 29 21:21:36.444612 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 29 21:21:36.444612 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 29 21:21:36.444612 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 29 21:21:36.444612 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 21:21:36.444612 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Aug 29 21:21:36.444612 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:21:36.444612 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 29 21:21:36.444612 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:21:36.444612 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:36.444612 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:36.444612 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:36.444612 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:36.444612 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:36.445845 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:36.445845 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:36.445845 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:36.445845 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:36.445845 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:36.445845 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:36.445845 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 29 21:21:36.445845 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 29 21:21:36.445845 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 29 21:21:36.445845 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 29 21:21:36.445845 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:36.445845 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:36.445845 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:36.445845 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:36.445845 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:36.445845 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:36.446911 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:36.446911 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:36.446911 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:36.446911 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:36.446911 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:36.446911 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:36.446911 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:36.446911 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:36.446911 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:36.446911 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:36.446911 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 29 21:21:36.446911 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 21:21:36.446911 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Aug 29 21:21:36.446911 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:21:36.449945 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 496/992] 10.222.0.90 () {56 vars in 1111 bytes} [Mon Aug 29 21:21:36 2022] GET /identity/v3/services/compute => generated 88 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:21:36.455428 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f56eb66d-57b5-4e20-94ce-8efab1023f3d None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:36.467175 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:36.559507 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f56eb66d-57b5-4e20-94ce-8efab1023f3d None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:36.596537 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfef3846-51af-49f9-94b9-c24d271af86b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:36.597162 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfef3846-51af-49f9-94b9-c24d271af86b None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:36.597446 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfef3846-51af-49f9-94b9-c24d271af86b None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:36.597724 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfef3846-51af-49f9-94b9-c24d271af86b None admin] PATH_INFO: `/v3/services` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:36.598397 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfef3846-51af-49f9-94b9-c24d271af86b None admin] RBAC: Adding query filter params (name=compute) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:36.598705 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfef3846-51af-49f9-94b9-c24d271af86b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:36.600072 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfef3846-51af-49f9-94b9-c24d271af86b None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:36.609907 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 497/993] 10.222.0.90 () {56 vars in 1112 bytes} [Mon Aug 29 21:21:36 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:36.611665 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6271090-b9cb-411d-97ac-0cccd082d826 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:36.656185 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-019d96ae-2efc-45f3-8f18-4c9bb96ca184 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:36.656470 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-019d96ae-2efc-45f3-8f18-4c9bb96ca184 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:36.656746 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-019d96ae-2efc-45f3-8f18-4c9bb96ca184 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:36.658686 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 498/994] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:21:36 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:21:36.661987 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:36.664712 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4819fa03-1696-4253-88c0-1cfd402ee2f1 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:36.665023 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4819fa03-1696-4253-88c0-1cfd402ee2f1 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:36.665313 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4819fa03-1696-4253-88c0-1cfd402ee2f1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:36.684079 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-4819fa03-1696-4253-88c0-1cfd402ee2f1 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:21:36.685350 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-4819fa03-1696-4253-88c0-1cfd402ee2f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:21:36.728277 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6271090-b9cb-411d-97ac-0cccd082d826 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:36.761947 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 499/995] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:21:36 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 29 21:21:36.769590 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a688572b-4a75-4a86-bc82-a588682a5166 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:36.770279 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a688572b-4a75-4a86-bc82-a588682a5166 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:36.770565 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a688572b-4a75-4a86-bc82-a588682a5166 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:36.770843 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a688572b-4a75-4a86-bc82-a588682a5166 None admin] PATH_INFO: `/v3/services` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:36.771484 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a688572b-4a75-4a86-bc82-a588682a5166 None admin] RBAC: Adding query filter params (type=compute) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:36.771790 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a688572b-4a75-4a86-bc82-a588682a5166 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:36.773158 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a688572b-4a75-4a86-bc82-a588682a5166 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:36.778452 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 497/996] 10.222.0.90 () {56 vars in 1112 bytes} [Mon Aug 29 21:21:36 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:21:36.784875 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4819fa03-1696-4253-88c0-1cfd402ee2f1 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:36.788365 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:36.895651 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4819fa03-1696-4253-88c0-1cfd402ee2f1 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:36.931949 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b36f37b5-0aee-40b9-9ce3-f71110e4c975 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:36.932567 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b36f37b5-0aee-40b9-9ce3-f71110e4c975 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:36.932853 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b36f37b5-0aee-40b9-9ce3-f71110e4c975 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:36.933131 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b36f37b5-0aee-40b9-9ce3-f71110e4c975 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:36.933861 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b36f37b5-0aee-40b9-9ce3-f71110e4c975 None admin] RBAC: Adding query filter params (interface=public, service_id=2c7f2f57683f400bb72a88b47a90a045, region_id=RegionOne) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:36.934179 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b36f37b5-0aee-40b9-9ce3-f71110e4c975 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:36.935546 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b36f37b5-0aee-40b9-9ce3-f71110e4c975 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:36.945374 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 500/997] 10.222.0.90 () {56 vars in 1251 bytes} [Mon Aug 29 21:21:36 2022] GET /identity/v3/endpoints?service_id=2c7f2f57683f400bb72a88b47a90a045&interface=public®ion_id=RegionOne => generated 194 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:21:36.946445 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a688572b-4a75-4a86-bc82-a588682a5166 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:37.027168 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b36f37b5-0aee-40b9-9ce3-f71110e4c975 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:37.052921 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a688572b-4a75-4a86-bc82-a588682a5166 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:37.060840 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:37.090365 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-687f07af-3ded-44ec-89db-ffe96337bd34 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:37.091019 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-687f07af-3ded-44ec-89db-ffe96337bd34 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:37.091299 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-687f07af-3ded-44ec-89db-ffe96337bd34 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:37.091578 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-687f07af-3ded-44ec-89db-ffe96337bd34 None admin] PATH_INFO: `/v3/services` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:37.092186 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-687f07af-3ded-44ec-89db-ffe96337bd34 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:37.093558 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-687f07af-3ded-44ec-89db-ffe96337bd34 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:37.098885 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:37.105821 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 498/998] 10.222.0.90 () {56 vars in 1087 bytes} [Mon Aug 29 21:21:36 2022] GET /identity/v3/services => generated 1727 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 29 21:21:37.148856 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b36f37b5-0aee-40b9-9ce3-f71110e4c975 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:37.187544 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff423547-41bc-463f-8106-89b9cb489684 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:37.188194 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff423547-41bc-463f-8106-89b9cb489684 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:37.189101 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff423547-41bc-463f-8106-89b9cb489684 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:37.189408 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff423547-41bc-463f-8106-89b9cb489684 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:37.190132 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff423547-41bc-463f-8106-89b9cb489684 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:37.191619 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff423547-41bc-463f-8106-89b9cb489684 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:37.201319 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-ff423547-41bc-463f-8106-89b9cb489684 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 29 21:21:37.201319 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:37.201319 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 29 21:21:37.201319 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 29 21:21:37.201319 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 29 21:21:37.201319 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 29 21:21:37.201319 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 29 21:21:37.201319 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 21:21:37.201319 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 29 21:21:37.201319 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:21:37.201319 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 29 21:21:37.201319 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:21:37.201319 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:37.201319 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:37.201319 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:37.201319 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:37.201319 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:37.202475 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:37.202475 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:37.202475 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:37.202475 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:37.202475 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:37.202475 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:37.202475 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 29 21:21:37.202475 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 29 21:21:37.202475 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 29 21:21:37.202475 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 29 21:21:37.202475 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:37.202475 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:37.202475 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:37.202475 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:37.202475 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:37.202475 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:37.203333 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:37.203333 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:37.203333 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:37.203333 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:37.203333 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:37.203333 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:37.203333 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:37.203333 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:37.203333 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:37.203333 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:37.203333 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 29 21:21:37.203333 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 21:21:37.203333 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 29 21:21:37.203333 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:21:37.208162 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 501/999] 10.222.0.90 () {56 vars in 1126 bytes} [Mon Aug 29 21:21:37 2022] GET /identity/v3/services/object-store => generated 93 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 29 21:21:37.211108 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-687f07af-3ded-44ec-89db-ffe96337bd34 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:37.318963 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-687f07af-3ded-44ec-89db-ffe96337bd34 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:37.353485 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a02de8b-6d18-4c9b-acf9-1a9f0ae3101e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:37.354141 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a02de8b-6d18-4c9b-acf9-1a9f0ae3101e None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:37.354420 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a02de8b-6d18-4c9b-acf9-1a9f0ae3101e None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:37.354694 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a02de8b-6d18-4c9b-acf9-1a9f0ae3101e None admin] PATH_INFO: `/v3/services` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:37.355334 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a02de8b-6d18-4c9b-acf9-1a9f0ae3101e None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:37.355640 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a02de8b-6d18-4c9b-acf9-1a9f0ae3101e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:37.357005 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a02de8b-6d18-4c9b-acf9-1a9f0ae3101e None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:37.363071 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 499/1000] 10.222.0.90 () {56 vars in 1122 bytes} [Mon Aug 29 21:21:37 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:37.368681 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff423547-41bc-463f-8106-89b9cb489684 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:37.472551 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff423547-41bc-463f-8106-89b9cb489684 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:37.501519 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:37.507953 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c464cebf-aa63-465d-be22-6fa58472be4f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:37.508572 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c464cebf-aa63-465d-be22-6fa58472be4f None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:37.508852 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c464cebf-aa63-465d-be22-6fa58472be4f None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:37.509122 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c464cebf-aa63-465d-be22-6fa58472be4f None admin] PATH_INFO: `/v3/services` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:37.509754 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c464cebf-aa63-465d-be22-6fa58472be4f None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:37.510092 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c464cebf-aa63-465d-be22-6fa58472be4f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:37.511456 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c464cebf-aa63-465d-be22-6fa58472be4f None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:37.517777 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 502/1001] 10.222.0.90 () {56 vars in 1122 bytes} [Mon Aug 29 21:21:37 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:37.520038 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-580d25a9-5770-4653-9233-e6ac74cf18c7 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:37.520501 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-580d25a9-5770-4653-9233-e6ac74cf18c7 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:37.520597 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-580d25a9-5770-4653-9233-e6ac74cf18c7 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:37.522191 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 500/1002] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:21:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:21:37.524199 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c464cebf-aa63-465d-be22-6fa58472be4f None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:37.527024 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-598fc0c6-34f5-4385-884f-6328658ded56 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:37.527313 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-598fc0c6-34f5-4385-884f-6328658ded56 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:37.527589 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-598fc0c6-34f5-4385-884f-6328658ded56 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:37.547386 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-598fc0c6-34f5-4385-884f-6328658ded56 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:21:37.548938 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-598fc0c6-34f5-4385-884f-6328658ded56 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:21:37.586788 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:37.637830 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 501/1003] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:21:37 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 29 21:21:37.638031 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c464cebf-aa63-465d-be22-6fa58472be4f None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:37.676563 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df434d13-d0ed-4295-ad14-469e7c73da69 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:37.677176 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df434d13-d0ed-4295-ad14-469e7c73da69 None admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:37.677456 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df434d13-d0ed-4295-ad14-469e7c73da69 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:37.677727 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df434d13-d0ed-4295-ad14-469e7c73da69 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:37.678441 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df434d13-d0ed-4295-ad14-469e7c73da69 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:37.679802 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df434d13-d0ed-4295-ad14-469e7c73da69 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:37.696049 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 503/1004] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:21:37 2022] POST /identity/v3/endpoints => generated 325 bytes in 172 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 21:21:37.911691 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-598fc0c6-34f5-4385-884f-6328658ded56 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:38.014454 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-598fc0c6-34f5-4385-884f-6328658ded56 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:38.055535 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc110e67-0816-4f9e-8a0e-960b7b3fcbf8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:38.056692 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc110e67-0816-4f9e-8a0e-960b7b3fcbf8 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:38.057173 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc110e67-0816-4f9e-8a0e-960b7b3fcbf8 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:38.057643 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc110e67-0816-4f9e-8a0e-960b7b3fcbf8 None admin] PATH_INFO: `/v3/services/image` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:38.058454 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc110e67-0816-4f9e-8a0e-960b7b3fcbf8 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:38.060119 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc110e67-0816-4f9e-8a0e-960b7b3fcbf8 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:38.069933 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-dc110e67-0816-4f9e-8a0e-960b7b3fcbf8 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Aug 29 21:21:38.069933 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:38.069933 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 29 21:21:38.069933 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 29 21:21:38.069933 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 29 21:21:38.069933 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 29 21:21:38.069933 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 29 21:21:38.069933 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 21:21:38.069933 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Aug 29 21:21:38.069933 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:21:38.069933 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 29 21:21:38.069933 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:21:38.069933 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:38.069933 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:38.069933 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:38.069933 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:38.069933 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:38.070969 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:38.070969 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:38.070969 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:38.070969 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:38.070969 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:38.070969 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:38.070969 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 29 21:21:38.070969 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 29 21:21:38.070969 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 29 21:21:38.070969 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 29 21:21:38.070969 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:38.070969 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:38.070969 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:38.070969 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:38.070969 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:38.070969 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:38.071825 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:38.071825 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:38.071825 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:38.071825 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:38.071825 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:38.071825 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:38.071825 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:38.071825 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:38.071825 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:38.071825 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:38.071825 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 29 21:21:38.071825 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 21:21:38.071825 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Aug 29 21:21:38.071825 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:21:38.074672 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 502/1005] 10.222.0.90 () {56 vars in 1105 bytes} [Mon Aug 29 21:21:37 2022] GET /identity/v3/services/image => generated 86 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:21:38.080961 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df434d13-d0ed-4295-ad14-469e7c73da69 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:38.140978 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:38.191158 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df434d13-d0ed-4295-ad14-469e7c73da69 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:38.228981 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f153e649-7554-4d37-be40-1741527927ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:38.229601 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f153e649-7554-4d37-be40-1741527927ee None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:38.229909 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f153e649-7554-4d37-be40-1741527927ee None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:38.230185 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f153e649-7554-4d37-be40-1741527927ee None admin] PATH_INFO: `/v3/services` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:38.230813 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f153e649-7554-4d37-be40-1741527927ee None admin] RBAC: Adding query filter params (name=image) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:38.231115 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f153e649-7554-4d37-be40-1741527927ee None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:38.232532 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f153e649-7554-4d37-be40-1741527927ee None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:38.237687 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 504/1006] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:21:38 2022] GET /identity/v3/services?name=image => generated 122 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:21:38.243147 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc110e67-0816-4f9e-8a0e-960b7b3fcbf8 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:38.270502 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:38.349554 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc110e67-0816-4f9e-8a0e-960b7b3fcbf8 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:38.383140 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cb65829-3f39-49cd-9d66-c06eba72d0bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:38.383754 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cb65829-3f39-49cd-9d66-c06eba72d0bc None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:38.384037 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cb65829-3f39-49cd-9d66-c06eba72d0bc None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:38.384311 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cb65829-3f39-49cd-9d66-c06eba72d0bc None admin] PATH_INFO: `/v3/services` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:38.384943 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cb65829-3f39-49cd-9d66-c06eba72d0bc None admin] RBAC: Adding query filter params (type=image) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:38.385248 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cb65829-3f39-49cd-9d66-c06eba72d0bc None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:38.386638 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cb65829-3f39-49cd-9d66-c06eba72d0bc None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:38.391794 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 503/1007] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:21:38 2022] GET /identity/v3/services?type=image => generated 122 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:39.056033 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0150e90f-0528-428c-811c-8e3d2858d002 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:39.056577 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0150e90f-0528-428c-811c-8e3d2858d002 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:39.056577 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0150e90f-0528-428c-811c-8e3d2858d002 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:39.058070 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 505/1008] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:21:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:21:39.063180 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac63d141-0982-4c61-ab89-b92d5c84f422 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:39.063717 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac63d141-0982-4c61-ab89-b92d5c84f422 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:39.063815 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac63d141-0982-4c61-ab89-b92d5c84f422 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:39.082543 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-ac63d141-0982-4c61-ab89-b92d5c84f422 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:21:39.083723 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-ac63d141-0982-4c61-ab89-b92d5c84f422 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:21:39.153107 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 504/1009] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:21:39 2022] POST /identity/v3/auth/tokens => generated 1717 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 21:21:39.418954 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0150e90f-0528-428c-811c-8e3d2858d002 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:39.528835 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0150e90f-0528-428c-811c-8e3d2858d002 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:39.550361 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:39.565569 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-676cf242-3ffb-44a3-8aa8-2473180dfee1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:39.566240 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-676cf242-3ffb-44a3-8aa8-2473180dfee1 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:39.566518 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-676cf242-3ffb-44a3-8aa8-2473180dfee1 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:39.566787 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-676cf242-3ffb-44a3-8aa8-2473180dfee1 None admin] PATH_INFO: `/v3/services/compute` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:39.567364 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-676cf242-3ffb-44a3-8aa8-2473180dfee1 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:39.568717 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-676cf242-3ffb-44a3-8aa8-2473180dfee1 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:39.577355 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-676cf242-3ffb-44a3-8aa8-2473180dfee1 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Aug 29 21:21:39.577355 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:39.577355 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 29 21:21:39.577355 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 29 21:21:39.577355 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 29 21:21:39.577355 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 29 21:21:39.577355 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 29 21:21:39.577355 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 21:21:39.577355 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Aug 29 21:21:39.577355 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:21:39.577355 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 29 21:21:39.577355 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:21:39.577355 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:39.577355 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:39.577355 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:39.577355 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:39.577355 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:39.578399 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:39.578399 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:39.578399 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:39.578399 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:39.578399 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:39.578399 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:39.578399 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 29 21:21:39.578399 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 29 21:21:39.578399 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 29 21:21:39.578399 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 29 21:21:39.578399 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:39.578399 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:39.578399 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:39.578399 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:39.578399 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:39.578399 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:39.579358 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:39.579358 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:39.579358 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:39.579358 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:39.579358 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:39.579358 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:39.579358 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:39.579358 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:39.579358 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:39.579358 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:39.579358 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 29 21:21:39.579358 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 21:21:39.579358 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Aug 29 21:21:39.579358 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:21:39.580764 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 506/1010] 10.222.0.90 () {56 vars in 1111 bytes} [Mon Aug 29 21:21:39 2022] GET /identity/v3/services/compute => generated 88 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:21:39.584427 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac63d141-0982-4c61-ab89-b92d5c84f422 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:39.603390 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:39.671418 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-315a30d9-ee46-4e0f-a003-8cb06ddb9b00 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:39.671602 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-315a30d9-ee46-4e0f-a003-8cb06ddb9b00 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:39.671918 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-315a30d9-ee46-4e0f-a003-8cb06ddb9b00 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:39.674350 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 507/1011] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:21:39 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 29 21:21:39.680062 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c810860-ccc2-43ec-9421-37646dba6cb7 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:39.680578 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c810860-ccc2-43ec-9421-37646dba6cb7 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:39.681043 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c810860-ccc2-43ec-9421-37646dba6cb7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:39.689068 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac63d141-0982-4c61-ab89-b92d5c84f422 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:39.701271 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-0c810860-ccc2-43ec-9421-37646dba6cb7 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:21:39.702793 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-0c810860-ccc2-43ec-9421-37646dba6cb7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:21:39.729202 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6900d59-e945-44ce-a447-8655714d3b1c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:39.729857 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6900d59-e945-44ce-a447-8655714d3b1c None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:39.730144 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6900d59-e945-44ce-a447-8655714d3b1c None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:39.730421 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6900d59-e945-44ce-a447-8655714d3b1c None admin] PATH_INFO: `/v3/services` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:39.731059 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6900d59-e945-44ce-a447-8655714d3b1c None admin] RBAC: Adding query filter params (name=compute) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:39.731368 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6900d59-e945-44ce-a447-8655714d3b1c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:39.732743 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6900d59-e945-44ce-a447-8655714d3b1c None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:39.742458 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 505/1012] 10.222.0.90 () {56 vars in 1112 bytes} [Mon Aug 29 21:21:39 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:39.743454 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6900d59-e945-44ce-a447-8655714d3b1c None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:39.783678 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 508/1013] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:21:39 2022] POST /identity/v3/auth/tokens => generated 1717 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 29 21:21:39.831914 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:39.854498 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6900d59-e945-44ce-a447-8655714d3b1c None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:39.890944 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdc33752-d009-49f0-9f07-7d658a0188b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:39.892126 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdc33752-d009-49f0-9f07-7d658a0188b0 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:39.892405 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdc33752-d009-49f0-9f07-7d658a0188b0 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:39.892676 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdc33752-d009-49f0-9f07-7d658a0188b0 None admin] PATH_INFO: `/v3/services` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:39.893307 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdc33752-d009-49f0-9f07-7d658a0188b0 None admin] RBAC: Adding query filter params (type=compute) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:39.893610 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdc33752-d009-49f0-9f07-7d658a0188b0 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:39.895000 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdc33752-d009-49f0-9f07-7d658a0188b0 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:39.904585 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 506/1014] 10.222.0.90 () {56 vars in 1112 bytes} [Mon Aug 29 21:21:39 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:21:39.905594 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c810860-ccc2-43ec-9421-37646dba6cb7 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:40.009472 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c810860-ccc2-43ec-9421-37646dba6cb7 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:40.024170 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:40.045650 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c1f8550-8039-40b4-a4ed-e5a886429592 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:40.046294 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c1f8550-8039-40b4-a4ed-e5a886429592 None admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:40.046578 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c1f8550-8039-40b4-a4ed-e5a886429592 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:40.046852 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c1f8550-8039-40b4-a4ed-e5a886429592 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:40.047546 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c1f8550-8039-40b4-a4ed-e5a886429592 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:40.048922 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c1f8550-8039-40b4-a4ed-e5a886429592 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:40.058322 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdc33752-d009-49f0-9f07-7d658a0188b0 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:40.071851 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 509/1015] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:21:39 2022] POST /identity/v3/endpoints => generated 334 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:21:40.181148 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdc33752-d009-49f0-9f07-7d658a0188b0 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:40.227188 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e835a63c-b013-4e52-bf80-3cdfa60be5f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:40.227814 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e835a63c-b013-4e52-bf80-3cdfa60be5f3 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:40.228097 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e835a63c-b013-4e52-bf80-3cdfa60be5f3 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:40.228375 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e835a63c-b013-4e52-bf80-3cdfa60be5f3 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:40.229519 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e835a63c-b013-4e52-bf80-3cdfa60be5f3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:40.230987 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e835a63c-b013-4e52-bf80-3cdfa60be5f3 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:40.232722 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 507/1016] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:21:40 2022] GET /identity/v3/domains/default => generated 199 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:40.238548 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c1f8550-8039-40b4-a4ed-e5a886429592 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:40.349739 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c1f8550-8039-40b4-a4ed-e5a886429592 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:40.357638 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0263573a-f170-4d63-a00d-adcf0b04280e None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:40.357983 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0263573a-f170-4d63-a00d-adcf0b04280e None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:40.358269 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0263573a-f170-4d63-a00d-adcf0b04280e None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:40.360028 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 508/1017] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:21:40 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 29 21:21:40.365409 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5efb5227-dc11-4594-b687-8abd87a062dc None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:40.365704 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5efb5227-dc11-4594-b687-8abd87a062dc None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:40.366051 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5efb5227-dc11-4594-b687-8abd87a062dc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:40.384777 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-5efb5227-dc11-4594-b687-8abd87a062dc None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:21:40.386027 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-5efb5227-dc11-4594-b687-8abd87a062dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:21:40.386811 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba2d8971-9987-4447-8023-11839be12ccb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:40.387624 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba2d8971-9987-4447-8023-11839be12ccb None admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:40.388076 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba2d8971-9987-4447-8023-11839be12ccb None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:40.388354 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba2d8971-9987-4447-8023-11839be12ccb None admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:40.389015 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba2d8971-9987-4447-8023-11839be12ccb None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:40.390334 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:40.390503 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba2d8971-9987-4447-8023-11839be12ccb None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:40.424933 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 510/1018] 10.222.0.90 () {60 vars in 1141 bytes} [Mon Aug 29 21:21:40 2022] POST /identity/v3/projects => generated 311 bytes in 187 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:21:40.476062 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 509/1019] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:21:40 2022] POST /identity/v3/auth/tokens => generated 1873 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 29 21:21:40.748648 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba2d8971-9987-4447-8023-11839be12ccb None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:40.775998 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:40.864109 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba2d8971-9987-4447-8023-11839be12ccb None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:40.900378 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a7d0151-104d-43bb-bf22-9aa40e991a9a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:40.900996 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a7d0151-104d-43bb-bf22-9aa40e991a9a None admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:40.901273 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a7d0151-104d-43bb-bf22-9aa40e991a9a None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:40.901542 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a7d0151-104d-43bb-bf22-9aa40e991a9a None admin] PATH_INFO: `/v3/services` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:40.902235 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a7d0151-104d-43bb-bf22-9aa40e991a9a None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:40.903600 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a7d0151-104d-43bb-bf22-9aa40e991a9a None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:40.921835 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 511/1020] 10.222.0.90 () {60 vars in 1141 bytes} [Mon Aug 29 21:21:40 2022] POST /identity/v3/services => generated 242 bytes in 172 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:21:42.115604 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e76e1778-79d6-4553-bd2f-d6b2a02f30d3 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:42.116095 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e76e1778-79d6-4553-bd2f-d6b2a02f30d3 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:42.116095 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e76e1778-79d6-4553-bd2f-d6b2a02f30d3 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:42.117536 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 510/1021] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:21:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:21:42.122397 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-688f71ed-d685-43ae-b684-d346bbde77ee None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:42.122778 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-688f71ed-d685-43ae-b684-d346bbde77ee None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:42.122969 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-688f71ed-d685-43ae-b684-d346bbde77ee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:42.140822 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-688f71ed-d685-43ae-b684-d346bbde77ee None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:21:42.142651 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-688f71ed-d685-43ae-b684-d346bbde77ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:21:42.216747 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 512/1022] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:21:42 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 21:21:42.392077 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2364d569-ee9e-482b-8bbb-354ba629e485 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:42.392265 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2364d569-ee9e-482b-8bbb-354ba629e485 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:42.392539 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2364d569-ee9e-482b-8bbb-354ba629e485 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:42.394047 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 511/1023] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:21:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:21:42.399241 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d34a9c0f-cbd8-4fbb-8912-eeb000d14c36 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:42.399532 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d34a9c0f-cbd8-4fbb-8912-eeb000d14c36 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:42.399809 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d34a9c0f-cbd8-4fbb-8912-eeb000d14c36 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:42.418705 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-d34a9c0f-cbd8-4fbb-8912-eeb000d14c36 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:21:42.419881 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-d34a9c0f-cbd8-4fbb-8912-eeb000d14c36 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:21:42.421895 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:42.482894 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2364d569-ee9e-482b-8bbb-354ba629e485 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:42.493500 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 513/1024] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:21:42 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 21:21:42.585931 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2364d569-ee9e-482b-8bbb-354ba629e485 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:42.620981 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8da83c6-a3fb-4758-95fa-7285366f74d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:42.621619 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8da83c6-a3fb-4758-95fa-7285366f74d2 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:42.621934 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8da83c6-a3fb-4758-95fa-7285366f74d2 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:42.622210 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8da83c6-a3fb-4758-95fa-7285366f74d2 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:42.629823 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:42.633939 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8da83c6-a3fb-4758-95fa-7285366f74d2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 21:21:42.638736 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-c8da83c6-a3fb-4758-95fa-7285366f74d2 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 29 21:21:42.638736 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:42.638736 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:42.638736 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:42.638736 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:42.638736 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:42.638736 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:42.638736 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:42.638736 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:42.638736 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:42.638736 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:42.638736 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:42.638736 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 21:21:42.638736 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 21:21:42.638736 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 21:21:42.638736 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 21:21:42.639654 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 21:21:42.639654 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 21:21:42.639654 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 21:21:42.639654 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 21:21:42.639654 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:42.639654 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:42.639654 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:42.639654 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:42.639654 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:42.639654 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:42.639654 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:42.639654 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:42.639654 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:42.639654 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:42.639654 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:42.639654 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:42.640523 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:42.640523 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:42.640523 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:42.640523 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:42.640523 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 21:21:42.640523 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 21:21:42.640523 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 21:21:42.640523 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 21:21:42.640523 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 21:21:42.640523 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 21:21:42.640523 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 29 21:21:42.640523 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:21:42.640523 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 512/1025] 10.222.0.90 () {56 vars in 1120 bytes} [Mon Aug 29 21:21:42 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 29 21:21:42.646078 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d34a9c0f-cbd8-4fbb-8912-eeb000d14c36 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:42.747491 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d34a9c0f-cbd8-4fbb-8912-eeb000d14c36 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:42.761953 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8da83c6-a3fb-4758-95fa-7285366f74d2 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:42.783010 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8aa7561-4bcc-4339-9281-358ed1ad10b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:42.783627 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8aa7561-4bcc-4339-9281-358ed1ad10b3 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:42.783903 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8aa7561-4bcc-4339-9281-358ed1ad10b3 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:42.784172 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8aa7561-4bcc-4339-9281-358ed1ad10b3 None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:42.784833 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8aa7561-4bcc-4339-9281-358ed1ad10b3 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:42.785188 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8aa7561-4bcc-4339-9281-358ed1ad10b3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:42.786580 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8aa7561-4bcc-4339-9281-358ed1ad10b3 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:42.791715 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:42.801306 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 514/1026] 10.222.0.90 () {56 vars in 1115 bytes} [Mon Aug 29 21:21:42 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:42.802424 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8aa7561-4bcc-4339-9281-358ed1ad10b3 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:42.873286 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8da83c6-a3fb-4758-95fa-7285366f74d2 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:42.909474 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-569e0ccd-5680-4855-8699-1759365c58b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:42.910069 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-569e0ccd-5680-4855-8699-1759365c58b9 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:42.910352 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-569e0ccd-5680-4855-8699-1759365c58b9 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:42.910628 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-569e0ccd-5680-4855-8699-1759365c58b9 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:42.911742 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-569e0ccd-5680-4855-8699-1759365c58b9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:42.913178 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-569e0ccd-5680-4855-8699-1759365c58b9 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:42.914797 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 513/1027] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:21:42 2022] GET /identity/v3/domains/default => generated 199 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:21:42.914930 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8aa7561-4bcc-4339-9281-358ed1ad10b3 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:42.919064 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ec85a6d-048b-4897-a407-9aede46a2e9b None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:42.919353 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ec85a6d-048b-4897-a407-9aede46a2e9b None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:42.919783 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ec85a6d-048b-4897-a407-9aede46a2e9b None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:42.920856 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 514/1028] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:21:42 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:21:42.921537 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ec85a6d-048b-4897-a407-9aede46a2e9b None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:42.949670 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9e81e0c-509c-4515-92bf-dc55b4378e2f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:42.950340 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9e81e0c-509c-4515-92bf-dc55b4378e2f None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:42.950621 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9e81e0c-509c-4515-92bf-dc55b4378e2f None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:42.950897 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9e81e0c-509c-4515-92bf-dc55b4378e2f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:42.962016 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9e81e0c-509c-4515-92bf-dc55b4378e2f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:42.963455 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9e81e0c-509c-4515-92bf-dc55b4378e2f None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:42.978450 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-d9e81e0c-509c-4515-92bf-dc55b4378e2f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:21:42.978450 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:42.978450 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:42.978450 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:42.978450 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:42.978450 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:42.978450 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:42.978450 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:42.978450 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:42.978450 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:42.978450 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:42.978450 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:42.978450 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 21:21:42.978450 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 21:21:42.978450 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 21:21:42.978450 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 21:21:42.979512 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:42.979512 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:42.979512 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:42.979512 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:42.979512 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:42.979512 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:42.979512 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:42.979512 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:42.979512 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:42.979512 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:42.979512 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:42.979512 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:42.979512 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:42.979512 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:42.979512 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:42.979512 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:42.979512 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 21:21:42.980495 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 21:21:42.980495 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:21:42.980495 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:21:42.980495 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 515/1029] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:21:42 2022] GET /identity/v3/domains/Default => generated 87 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:21:42.980925 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ca0670-0e4a-4afe-bec5-e2dc5b6f04d2 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:42.980999 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ca0670-0e4a-4afe-bec5-e2dc5b6f04d2 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:42.981252 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ca0670-0e4a-4afe-bec5-e2dc5b6f04d2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:42.994823 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:42.999498 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-e8ca0670-0e4a-4afe-bec5-e2dc5b6f04d2 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:21:43.000916 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-e8ca0670-0e4a-4afe-bec5-e2dc5b6f04d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:21:43.029169 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ec85a6d-048b-4897-a407-9aede46a2e9b None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:43.048238 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:43.064727 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01e447e2-bf99-4b9d-9ba3-cdcbb37007e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:43.065344 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01e447e2-bf99-4b9d-9ba3-cdcbb37007e4 None admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:43.065625 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01e447e2-bf99-4b9d-9ba3-cdcbb37007e4 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:43.065940 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01e447e2-bf99-4b9d-9ba3-cdcbb37007e4 None admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:43.066593 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01e447e2-bf99-4b9d-9ba3-cdcbb37007e4 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:43.068048 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01e447e2-bf99-4b9d-9ba3-cdcbb37007e4 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:43.068639 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 516/1030] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:21:42 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 21:21:43.069284 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8ca0670-0e4a-4afe-bec5-e2dc5b6f04d2 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:43.093419 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 515/1031] 10.222.0.90 () {60 vars in 1132 bytes} [Mon Aug 29 21:21:42 2022] POST /identity/v3/users => generated 285 bytes in 172 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:21:43.187654 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8ca0670-0e4a-4afe-bec5-e2dc5b6f04d2 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:43.226277 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b0b46db-d86d-4a55-aae5-179d746711ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:43.226900 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b0b46db-d86d-4a55-aae5-179d746711ad None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:43.227182 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b0b46db-d86d-4a55-aae5-179d746711ad None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:43.227453 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b0b46db-d86d-4a55-aae5-179d746711ad None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:43.228103 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b0b46db-d86d-4a55-aae5-179d746711ad None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:43.228405 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b0b46db-d86d-4a55-aae5-179d746711ad None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:43.229812 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b0b46db-d86d-4a55-aae5-179d746711ad None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:43.243182 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 517/1032] 10.222.0.90 () {56 vars in 1109 bytes} [Mon Aug 29 21:21:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:43.249100 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01e447e2-bf99-4b9d-9ba3-cdcbb37007e4 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:43.343106 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b0b46db-d86d-4a55-aae5-179d746711ad None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:43.363677 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01e447e2-bf99-4b9d-9ba3-cdcbb37007e4 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:43.388012 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:43.388333 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:43.402470 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dbfb1f2-ce87-4259-a902-c885033f8a64 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:43.403119 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dbfb1f2-ce87-4259-a902-c885033f8a64 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:43.403400 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dbfb1f2-ce87-4259-a902-c885033f8a64 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:43.403666 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dbfb1f2-ce87-4259-a902-c885033f8a64 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:43.411863 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dbfb1f2-ce87-4259-a902-c885033f8a64 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:43.413324 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dbfb1f2-ce87-4259-a902-c885033f8a64 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:43.425594 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-7dbfb1f2-ce87-4259-a902-c885033f8a64 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 29 21:21:43.425594 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:43.425594 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:43.425594 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:43.425594 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:43.425594 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:43.425594 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:43.425594 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:43.425594 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:43.425594 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:43.425594 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:43.425594 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:43.425594 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 29 21:21:43.425594 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 29 21:21:43.425594 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 29 21:21:43.425594 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 29 21:21:43.426657 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:43.426657 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:43.426657 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 29 21:21:43.426657 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 21:21:43.426657 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 29 21:21:43.426657 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 21:21:43.426657 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:43.426657 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:43.426657 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:43.426657 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:43.426657 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:43.426657 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:43.426657 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:43.426657 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:43.426657 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:43.426657 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:43.426657 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:43.426657 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:43.427669 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:43.427669 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:43.427669 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 29 21:21:43.427669 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 29 21:21:43.427669 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 29 21:21:43.427669 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 29 21:21:43.427669 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 29 21:21:43.427669 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 29 21:21:43.427669 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 29 21:21:43.427669 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:21:43.427669 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 516/1033] 10.222.0.90 () {56 vars in 1093 bytes} [Mon Aug 29 21:21:43 2022] GET /identity/v3/users/nova => generated 82 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 29 21:21:43.432482 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dbfb1f2-ce87-4259-a902-c885033f8a64 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:43.453261 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b0b46db-d86d-4a55-aae5-179d746711ad None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:43.492605 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb34a2ac-ca28-43d1-8d5f-cbebf26f1212 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:43.493284 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb34a2ac-ca28-43d1-8d5f-cbebf26f1212 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:43.493563 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb34a2ac-ca28-43d1-8d5f-cbebf26f1212 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:43.493860 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb34a2ac-ca28-43d1-8d5f-cbebf26f1212 None admin] PATH_INFO: `/v3/services/image` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:43.494449 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb34a2ac-ca28-43d1-8d5f-cbebf26f1212 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:43.495814 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb34a2ac-ca28-43d1-8d5f-cbebf26f1212 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:43.505755 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-cb34a2ac-ca28-43d1-8d5f-cbebf26f1212 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Aug 29 21:21:43.505755 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:43.505755 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 29 21:21:43.505755 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 29 21:21:43.505755 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 29 21:21:43.505755 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 29 21:21:43.505755 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 29 21:21:43.505755 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 21:21:43.505755 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Aug 29 21:21:43.505755 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:21:43.505755 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 29 21:21:43.505755 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:21:43.505755 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:43.505755 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:43.505755 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:43.505755 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:43.505755 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:43.506906 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:43.506906 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:43.506906 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:43.506906 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:43.506906 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:43.506906 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:43.506906 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 29 21:21:43.506906 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 29 21:21:43.506906 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 29 21:21:43.506906 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 29 21:21:43.506906 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:43.506906 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:43.506906 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:43.506906 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:43.506906 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:43.506906 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:43.507898 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:43.507898 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:43.507898 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:43.507898 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:43.507898 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:43.507898 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:43.507898 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:43.507898 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:43.507898 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:43.507898 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:43.507898 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 29 21:21:43.507898 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 21:21:43.507898 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Aug 29 21:21:43.507898 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:21:43.507898 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 518/1034] 10.222.0.90 () {56 vars in 1105 bytes} [Mon Aug 29 21:21:43 2022] GET /identity/v3/services/image => generated 86 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 29 21:21:43.513668 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb34a2ac-ca28-43d1-8d5f-cbebf26f1212 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:43.546651 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dbfb1f2-ce87-4259-a902-c885033f8a64 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:43.583960 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ea44974-977f-4646-8843-47fcf943c4cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:43.584563 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ea44974-977f-4646-8843-47fcf943c4cf None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:43.584853 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ea44974-977f-4646-8843-47fcf943c4cf None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:43.585126 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ea44974-977f-4646-8843-47fcf943c4cf None admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:43.585906 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ea44974-977f-4646-8843-47fcf943c4cf None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:43.586212 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ea44974-977f-4646-8843-47fcf943c4cf None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:43.587640 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ea44974-977f-4646-8843-47fcf943c4cf None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:43.606494 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 517/1035] 10.222.0.90 () {56 vars in 1133 bytes} [Mon Aug 29 21:21:43 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:43.612193 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ea44974-977f-4646-8843-47fcf943c4cf None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:43.634256 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb34a2ac-ca28-43d1-8d5f-cbebf26f1212 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:43.644923 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:43.661286 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:43.679436 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04d92583-ab48-4457-b576-03ee54273715 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:43.680205 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04d92583-ab48-4457-b576-03ee54273715 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:43.680794 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04d92583-ab48-4457-b576-03ee54273715 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:43.681078 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04d92583-ab48-4457-b576-03ee54273715 None admin] PATH_INFO: `/v3/services` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:43.682031 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04d92583-ab48-4457-b576-03ee54273715 None admin] RBAC: Adding query filter params (name=image) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:43.682341 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04d92583-ab48-4457-b576-03ee54273715 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:43.684450 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04d92583-ab48-4457-b576-03ee54273715 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:43.691460 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 519/1036] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:21:43 2022] GET /identity/v3/services?name=image => generated 122 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:43.697221 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04d92583-ab48-4457-b576-03ee54273715 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:43.727692 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ea44974-977f-4646-8843-47fcf943c4cf None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:43.763608 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e1b9253-3a04-46be-9f45-078194446934 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:43.764498 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e1b9253-3a04-46be-9f45-078194446934 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:43.764941 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e1b9253-3a04-46be-9f45-078194446934 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:43.765479 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e1b9253-3a04-46be-9f45-078194446934 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:43.811053 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04d92583-ab48-4457-b576-03ee54273715 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:43.842901 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e1b9253-3a04-46be-9f45-078194446934 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:43.844370 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e1b9253-3a04-46be-9f45-078194446934 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:43.851332 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-057b8514-01ea-445d-8c49-6add429ccad0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:43.851952 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-057b8514-01ea-445d-8c49-6add429ccad0 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:43.852232 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-057b8514-01ea-445d-8c49-6add429ccad0 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:43.852504 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-057b8514-01ea-445d-8c49-6add429ccad0 None admin] PATH_INFO: `/v3/services` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:43.853137 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-057b8514-01ea-445d-8c49-6add429ccad0 None admin] RBAC: Adding query filter params (type=image) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:43.853440 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-057b8514-01ea-445d-8c49-6add429ccad0 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:43.857685 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-057b8514-01ea-445d-8c49-6add429ccad0 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:43.867799 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 520/1037] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:21:43 2022] GET /identity/v3/services?type=image => generated 350 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:21:43.868819 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-057b8514-01ea-445d-8c49-6add429ccad0 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:43.895026 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:43.908861 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:43.954960 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 518/1038] 10.222.0.90 () {58 vars in 1282 bytes} [Mon Aug 29 21:21:43 2022] GET /identity/v3/auth/tokens => generated 1969 bytes in 343 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Aug 29 21:21:43.960590 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e1b9253-3a04-46be-9f45-078194446934 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:43.991655 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-057b8514-01ea-445d-8c49-6add429ccad0 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:44.028286 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef5a7203-d9cf-497f-aa3d-ab47b3e9e159 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:44.028899 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef5a7203-d9cf-497f-aa3d-ab47b3e9e159 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:44.029182 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef5a7203-d9cf-497f-aa3d-ab47b3e9e159 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:44.029455 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef5a7203-d9cf-497f-aa3d-ab47b3e9e159 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:44.030317 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef5a7203-d9cf-497f-aa3d-ab47b3e9e159 None admin] RBAC: Adding query filter params (interface=public, service_id=d41e3137f0c544568142a693b6f828fc, region_id=RegionOne) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:44.030623 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef5a7203-d9cf-497f-aa3d-ab47b3e9e159 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:44.032737 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef5a7203-d9cf-497f-aa3d-ab47b3e9e159 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:44.043263 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 521/1039] 10.222.0.90 () {56 vars in 1251 bytes} [Mon Aug 29 21:21:43 2022] GET /identity/v3/endpoints?service_id=d41e3137f0c544568142a693b6f828fc&interface=public®ion_id=RegionOne => generated 194 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:21:44.044213 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef5a7203-d9cf-497f-aa3d-ab47b3e9e159 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:44.072257 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e1b9253-3a04-46be-9f45-078194446934 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:44.109597 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45d754a1-c0d7-45c0-be84-989afe87d125 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:44.110936 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45d754a1-c0d7-45c0-be84-989afe87d125 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:44.111214 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45d754a1-c0d7-45c0-be84-989afe87d125 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:44.111491 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45d754a1-c0d7-45c0-be84-989afe87d125 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:44.122856 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45d754a1-c0d7-45c0-be84-989afe87d125 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:44.124316 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45d754a1-c0d7-45c0-be84-989afe87d125 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:44.126919 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:44.143626 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-45d754a1-c0d7-45c0-be84-989afe87d125 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:21:44.143626 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:44.143626 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:44.143626 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:44.143626 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:44.143626 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:44.143626 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:44.143626 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:44.143626 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:44.143626 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:44.143626 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:44.143626 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:44.143626 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 21:21:44.143626 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 21:21:44.143626 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 21:21:44.143626 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 21:21:44.144662 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:44.144662 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:44.144662 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:44.144662 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:44.144662 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:44.144662 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:44.144662 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:44.144662 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:44.144662 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:44.144662 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:44.144662 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:44.144662 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:44.144662 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:44.144662 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:44.144662 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:44.144662 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:44.144662 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 21:21:44.145606 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 21:21:44.145606 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:21:44.145606 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:21:44.149622 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 519/1040] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:21:43 2022] GET /identity/v3/domains/Default => generated 87 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 29 21:21:44.150455 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45d754a1-c0d7-45c0-be84-989afe87d125 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:44.151726 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:44.155964 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef5a7203-d9cf-497f-aa3d-ab47b3e9e159 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:44.194423 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fc54469-99ea-40e6-8740-6c6fc5fc982d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:44.195068 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fc54469-99ea-40e6-8740-6c6fc5fc982d None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:44.195638 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fc54469-99ea-40e6-8740-6c6fc5fc982d None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:44.195638 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fc54469-99ea-40e6-8740-6c6fc5fc982d None admin] PATH_INFO: `/v3/services` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:44.196200 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fc54469-99ea-40e6-8740-6c6fc5fc982d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:44.197571 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fc54469-99ea-40e6-8740-6c6fc5fc982d None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:44.213819 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 522/1041] 10.222.0.90 () {56 vars in 1087 bytes} [Mon Aug 29 21:21:44 2022] GET /identity/v3/services => generated 1957 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:21:44.275480 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45d754a1-c0d7-45c0-be84-989afe87d125 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:44.313075 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a70085f1-21fb-42b8-b187-70740a4591d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:44.313716 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a70085f1-21fb-42b8-b187-70740a4591d2 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:44.314021 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a70085f1-21fb-42b8-b187-70740a4591d2 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:44.314299 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a70085f1-21fb-42b8-b187-70740a4591d2 None admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:44.314961 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a70085f1-21fb-42b8-b187-70740a4591d2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:44.315258 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a70085f1-21fb-42b8-b187-70740a4591d2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:44.316660 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a70085f1-21fb-42b8-b187-70740a4591d2 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:44.334915 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 520/1042] 10.222.0.90 () {56 vars in 1109 bytes} [Mon Aug 29 21:21:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:44.336247 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fc54469-99ea-40e6-8740-6c6fc5fc982d None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:44.444993 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fc54469-99ea-40e6-8740-6c6fc5fc982d None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:44.481870 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b618530f-660f-4198-a799-eae512cad516 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:44.482828 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b618530f-660f-4198-a799-eae512cad516 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:44.483296 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b618530f-660f-4198-a799-eae512cad516 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:44.483752 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b618530f-660f-4198-a799-eae512cad516 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:44.491209 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b618530f-660f-4198-a799-eae512cad516 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:44.492838 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b618530f-660f-4198-a799-eae512cad516 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:44.504448 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-b618530f-660f-4198-a799-eae512cad516 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 21:21:44.504448 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:44.504448 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:44.504448 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:44.504448 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:44.504448 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:44.504448 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:44.504448 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:44.504448 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:44.504448 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:44.504448 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:44.504448 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:44.504448 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 29 21:21:44.504448 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 29 21:21:44.504448 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 29 21:21:44.505391 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 29 21:21:44.505391 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:44.505391 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:44.505391 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:44.505391 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:44.505391 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:44.505391 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:44.505391 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:44.505391 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:44.505391 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:44.505391 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:44.505391 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:44.505391 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:44.505391 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:44.505391 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:44.505391 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:44.505391 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:44.506361 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 29 21:21:44.506361 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 29 21:21:44.506361 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 29 21:21:44.506361 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 29 21:21:44.506361 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 29 21:21:44.506361 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 29 21:21:44.506361 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 21:21:44.506361 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:21:44.507597 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 523/1043] 10.222.0.90 () {56 vars in 1111 bytes} [Mon Aug 29 21:21:44 2022] GET /identity/v3/projects/service => generated 88 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:21:44.513384 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a70085f1-21fb-42b8-b187-70740a4591d2 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:44.574216 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:44.622031 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a70085f1-21fb-42b8-b187-70740a4591d2 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:44.657712 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6fead4f-dd60-45ef-8179-4ffd495958d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:44.658370 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6fead4f-dd60-45ef-8179-4ffd495958d3 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:44.658654 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6fead4f-dd60-45ef-8179-4ffd495958d3 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:44.658929 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6fead4f-dd60-45ef-8179-4ffd495958d3 None admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:44.659627 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6fead4f-dd60-45ef-8179-4ffd495958d3 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:44.659934 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6fead4f-dd60-45ef-8179-4ffd495958d3 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:44.662969 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6fead4f-dd60-45ef-8179-4ffd495958d3 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:44.674451 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 521/1044] 10.222.0.90 () {56 vars in 1148 bytes} [Mon Aug 29 21:21:44 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:44.680173 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b618530f-660f-4198-a799-eae512cad516 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:44.707229 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:44.787675 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b618530f-660f-4198-a799-eae512cad516 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:44.822196 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5eaa817-7c84-44f9-8718-a24b1a7deb82 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:44.822807 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5eaa817-7c84-44f9-8718-a24b1a7deb82 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:44.823088 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5eaa817-7c84-44f9-8718-a24b1a7deb82 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:44.823365 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5eaa817-7c84-44f9-8718-a24b1a7deb82 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:44.824092 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5eaa817-7c84-44f9-8718-a24b1a7deb82 None admin] RBAC: Adding query filter params (role.id=9c919ca974684bc492e75274b81c78ad, scope.project.id=ec03e112e3d547fbab6faf7b485c3fca, user.id=a35b5b7dea7b421da056f063d2436ffb) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:44.824453 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5eaa817-7c84-44f9-8718-a24b1a7deb82 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:44.825992 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5eaa817-7c84-44f9-8718-a24b1a7deb82 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:44.837822 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 524/1045] 10.222.0.90 () {56 vars in 1374 bytes} [Mon Aug 29 21:21:44 2022] GET /identity/v3/role_assignments?user.id=a35b5b7dea7b421da056f063d2436ffb&scope.project.id=ec03e112e3d547fbab6faf7b485c3fca&role.id=9c919ca974684bc492e75274b81c78ad => generated 259 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:45.113819 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b6259bd-13a2-4d78-87da-cd77bde403e7 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:45.114119 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b6259bd-13a2-4d78-87da-cd77bde403e7 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:45.114323 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b6259bd-13a2-4d78-87da-cd77bde403e7 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:45.115841 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 522/1046] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:21:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:21:45.120758 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0270c36-de30-433b-93d7-a82bca84b7ef None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:45.121055 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0270c36-de30-433b-93d7-a82bca84b7ef None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:45.121341 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0270c36-de30-433b-93d7-a82bca84b7ef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:45.140683 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-b0270c36-de30-433b-93d7-a82bca84b7ef None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:21:45.142379 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-b0270c36-de30-433b-93d7-a82bca84b7ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:21:45.210847 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 525/1047] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:21:45 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 21:21:45.477704 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b6259bd-13a2-4d78-87da-cd77bde403e7 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:45.586661 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b6259bd-13a2-4d78-87da-cd77bde403e7 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:45.601240 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:45.623363 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33208c10-5900-4fb9-a8d1-65b32fc88a10 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:45.624011 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33208c10-5900-4fb9-a8d1-65b32fc88a10 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:45.624297 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33208c10-5900-4fb9-a8d1-65b32fc88a10 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:45.624575 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33208c10-5900-4fb9-a8d1-65b32fc88a10 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:45.636573 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33208c10-5900-4fb9-a8d1-65b32fc88a10 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 21:21:45.641469 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-33208c10-5900-4fb9-a8d1-65b32fc88a10 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 21:21:45.641469 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:45.641469 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:45.641469 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:45.641469 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:45.641469 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:45.641469 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:45.641469 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:45.641469 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:45.641469 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:45.641469 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:45.641469 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:45.641469 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 21:21:45.641469 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 21:21:45.641469 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 21:21:45.641469 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 21:21:45.642594 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 21:21:45.642594 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 21:21:45.642594 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 21:21:45.642594 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 21:21:45.642594 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:45.642594 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:45.642594 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:45.642594 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:45.642594 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:45.642594 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:45.642594 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:45.642594 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:45.642594 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:45.642594 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:45.642594 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:45.642594 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:45.643974 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:45.643974 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:45.643974 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:45.643974 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:45.643974 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 21:21:45.643974 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 21:21:45.643974 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 21:21:45.643974 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 21:21:45.643974 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 21:21:45.643974 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 21:21:45.643974 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 21:21:45.643974 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:21:45.646137 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 523/1048] 10.222.0.90 () {56 vars in 1096 bytes} [Mon Aug 29 21:21:45 2022] GET /identity/v3/roles/admin => generated 83 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:21:45.648534 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0270c36-de30-433b-93d7-a82bca84b7ef None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:45.666502 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:45.754000 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0270c36-de30-433b-93d7-a82bca84b7ef None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:45.789744 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e25bfa18-5d14-400b-85d9-25fd9f251e43 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:45.790385 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e25bfa18-5d14-400b-85d9-25fd9f251e43 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:45.790662 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e25bfa18-5d14-400b-85d9-25fd9f251e43 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:45.790980 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e25bfa18-5d14-400b-85d9-25fd9f251e43 None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:45.791615 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e25bfa18-5d14-400b-85d9-25fd9f251e43 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:45.791912 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e25bfa18-5d14-400b-85d9-25fd9f251e43 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:45.793282 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e25bfa18-5d14-400b-85d9-25fd9f251e43 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:45.802299 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 526/1049] 10.222.0.90 () {56 vars in 1099 bytes} [Mon Aug 29 21:21:45 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:45.807783 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33208c10-5900-4fb9-a8d1-65b32fc88a10 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:45.921956 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33208c10-5900-4fb9-a8d1-65b32fc88a10 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:45.958434 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86910fe5-430f-41c2-8444-abfbaaf327d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:45.959071 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86910fe5-430f-41c2-8444-abfbaaf327d8 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:45.959352 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86910fe5-430f-41c2-8444-abfbaaf327d8 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:45.959627 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86910fe5-430f-41c2-8444-abfbaaf327d8 None admin] PATH_INFO: `/v3/users/f4b0f296bcd24afcb7d999777a5821e5` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:45.973829 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86910fe5-430f-41c2-8444-abfbaaf327d8 None admin] RBAC: Authorizing `identity:get_user(user_id=f4b0f296bcd24afcb7d999777a5821e5)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:45.975462 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86910fe5-430f-41c2-8444-abfbaaf327d8 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:45.982206 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 524/1050] 10.222.0.90 () {56 vars in 1177 bytes} [Mon Aug 29 21:21:45 2022] GET /identity/v3/users/f4b0f296bcd24afcb7d999777a5821e5 => generated 285 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:45.983192 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e25bfa18-5d14-400b-85d9-25fd9f251e43 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:46.065030 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:46.087914 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e25bfa18-5d14-400b-85d9-25fd9f251e43 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:46.123554 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77177ede-c753-44ba-a823-4ef18f0911f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:46.124171 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77177ede-c753-44ba-a823-4ef18f0911f7 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:46.124460 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77177ede-c753-44ba-a823-4ef18f0911f7 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:46.124732 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77177ede-c753-44ba-a823-4ef18f0911f7 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:46.143708 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b14c8af-1d30-4b0b-becb-f25463ac43df None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:46.143989 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b14c8af-1d30-4b0b-becb-f25463ac43df None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:46.144266 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b14c8af-1d30-4b0b-becb-f25463ac43df None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:46.145749 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 525/1051] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:21:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 29 21:21:46.150665 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35892f2f-20bb-4f6a-b1d3-762643c8dfdf None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:46.150950 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35892f2f-20bb-4f6a-b1d3-762643c8dfdf None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:46.151350 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35892f2f-20bb-4f6a-b1d3-762643c8dfdf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:46.169766 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-35892f2f-20bb-4f6a-b1d3-762643c8dfdf None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:21:46.171011 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-35892f2f-20bb-4f6a-b1d3-762643c8dfdf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:21:46.197736 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:46.201006 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77177ede-c753-44ba-a823-4ef18f0911f7 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:46.202502 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77177ede-c753-44ba-a823-4ef18f0911f7 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:46.250100 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 526/1052] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:21:46 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 29 21:21:46.282115 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:46.305621 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 527/1053] 10.222.0.90 () {58 vars in 1282 bytes} [Mon Aug 29 21:21:45 2022] GET /identity/v3/auth/tokens => generated 1969 bytes in 323 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Aug 29 21:21:46.311336 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35892f2f-20bb-4f6a-b1d3-762643c8dfdf None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:46.415337 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35892f2f-20bb-4f6a-b1d3-762643c8dfdf None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:46.450735 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d73a2a3-0a8e-42f5-b07a-54c09b544999 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:46.451374 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d73a2a3-0a8e-42f5-b07a-54c09b544999 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:46.451653 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d73a2a3-0a8e-42f5-b07a-54c09b544999 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:46.451927 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d73a2a3-0a8e-42f5-b07a-54c09b544999 None admin] PATH_INFO: `/v3/projects/4e31c4f75dea41809073912235f7ab55` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:46.453039 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d73a2a3-0a8e-42f5-b07a-54c09b544999 None admin] RBAC: Authorizing `identity:get_project(project_id=4e31c4f75dea41809073912235f7ab55)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:46.455130 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d73a2a3-0a8e-42f5-b07a-54c09b544999 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:46.457080 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 527/1054] 10.222.0.90 () {56 vars in 1186 bytes} [Mon Aug 29 21:21:46 2022] GET /identity/v3/projects/4e31c4f75dea41809073912235f7ab55 => generated 311 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:46.462889 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77177ede-c753-44ba-a823-4ef18f0911f7 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:46.524009 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d73a2a3-0a8e-42f5-b07a-54c09b544999 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:46.556686 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:46.566705 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77177ede-c753-44ba-a823-4ef18f0911f7 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:46.605517 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5906f416-2dfd-41e3-ac28-a09519ea41ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:46.606441 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5906f416-2dfd-41e3-ac28-a09519ea41ff None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:46.606724 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5906f416-2dfd-41e3-ac28-a09519ea41ff None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:46.607000 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5906f416-2dfd-41e3-ac28-a09519ea41ff None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:46.607727 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5906f416-2dfd-41e3-ac28-a09519ea41ff None admin] RBAC: Adding query filter params (role.id=a6e562cdc96c4be691d638990f8403fd, scope.project.id=4e31c4f75dea41809073912235f7ab55, user.id=f4b0f296bcd24afcb7d999777a5821e5) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:46.608037 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5906f416-2dfd-41e3-ac28-a09519ea41ff None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:46.609885 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5906f416-2dfd-41e3-ac28-a09519ea41ff None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:46.615700 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 528/1055] 10.222.0.90 () {56 vars in 1374 bytes} [Mon Aug 29 21:21:46 2022] GET /identity/v3/role_assignments?user.id=f4b0f296bcd24afcb7d999777a5821e5&scope.project.id=4e31c4f75dea41809073912235f7ab55&role.id=a6e562cdc96c4be691d638990f8403fd => generated 259 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:21:46.649492 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d73a2a3-0a8e-42f5-b07a-54c09b544999 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:46.688901 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eea23db6-9cee-448b-b311-e5975a52dff8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:46.689510 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea23db6-9cee-448b-b311-e5975a52dff8 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:46.689819 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea23db6-9cee-448b-b311-e5975a52dff8 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:46.690101 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea23db6-9cee-448b-b311-e5975a52dff8 None admin] PATH_INFO: `/v3/services/image` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:46.690707 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eea23db6-9cee-448b-b311-e5975a52dff8 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:46.692112 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eea23db6-9cee-448b-b311-e5975a52dff8 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:46.701839 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-eea23db6-9cee-448b-b311-e5975a52dff8 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Aug 29 21:21:46.701839 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:46.701839 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 29 21:21:46.701839 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 29 21:21:46.701839 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 29 21:21:46.701839 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 29 21:21:46.701839 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 29 21:21:46.701839 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 21:21:46.701839 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Aug 29 21:21:46.701839 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:21:46.701839 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 29 21:21:46.701839 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:21:46.701839 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:46.701839 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:46.701839 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:46.701839 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:46.701839 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:46.702961 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:46.702961 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:46.702961 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:46.702961 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:46.702961 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:46.702961 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:46.702961 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 29 21:21:46.702961 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 29 21:21:46.702961 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 29 21:21:46.702961 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 29 21:21:46.702961 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:46.702961 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:46.702961 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:46.702961 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:46.702961 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:46.702961 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:46.704469 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:46.704469 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:46.704469 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:46.704469 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:46.704469 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:46.704469 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:46.704469 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:46.704469 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:46.704469 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:46.704469 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:46.704469 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 29 21:21:46.704469 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 21:21:46.704469 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Aug 29 21:21:46.704469 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:21:46.708757 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 528/1056] 10.222.0.90 () {56 vars in 1105 bytes} [Mon Aug 29 21:21:46 2022] GET /identity/v3/services/image => generated 86 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 29 21:21:46.710124 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5906f416-2dfd-41e3-ac28-a09519ea41ff None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:46.759854 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:46.784323 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c75b198-50df-4ea9-8eda-53522fc9f552 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:46.784621 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c75b198-50df-4ea9-8eda-53522fc9f552 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:46.784897 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c75b198-50df-4ea9-8eda-53522fc9f552 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:46.786290 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 529/1057] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:21:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 29 21:21:46.791182 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d32f5f72-ce9d-4cfd-8ac2-27d30edfa3b4 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:46.791473 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d32f5f72-ce9d-4cfd-8ac2-27d30edfa3b4 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:46.791758 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d32f5f72-ce9d-4cfd-8ac2-27d30edfa3b4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:46.810127 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-d32f5f72-ce9d-4cfd-8ac2-27d30edfa3b4 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:21:46.811324 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-d32f5f72-ce9d-4cfd-8ac2-27d30edfa3b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:21:46.832349 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5906f416-2dfd-41e3-ac28-a09519ea41ff None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:46.873899 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7718adbc-a860-403a-94e7-a25def20a519 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:46.874517 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7718adbc-a860-403a-94e7-a25def20a519 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:46.874796 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7718adbc-a860-403a-94e7-a25def20a519 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:46.875069 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7718adbc-a860-403a-94e7-a25def20a519 None admin] PATH_INFO: `/v3/services` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:46.875752 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7718adbc-a860-403a-94e7-a25def20a519 None admin] RBAC: Adding query filter params (name=image) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:46.876004 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7718adbc-a860-403a-94e7-a25def20a519 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:46.877368 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7718adbc-a860-403a-94e7-a25def20a519 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:46.882234 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 529/1058] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:21:46 2022] GET /identity/v3/services?name=image => generated 122 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:46.887693 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7718adbc-a860-403a-94e7-a25def20a519 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:46.897843 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 530/1059] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:21:46 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 29 21:21:46.990863 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7718adbc-a860-403a-94e7-a25def20a519 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:46.997685 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:47.026264 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9461687b-676d-4ab6-aa36-4ceec2d16e1a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:47.026879 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9461687b-676d-4ab6-aa36-4ceec2d16e1a None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:47.027162 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9461687b-676d-4ab6-aa36-4ceec2d16e1a None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:47.027434 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9461687b-676d-4ab6-aa36-4ceec2d16e1a None admin] PATH_INFO: `/v3/services` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:47.028067 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9461687b-676d-4ab6-aa36-4ceec2d16e1a None admin] RBAC: Adding query filter params (type=image) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:47.028369 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9461687b-676d-4ab6-aa36-4ceec2d16e1a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:47.029733 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9461687b-676d-4ab6-aa36-4ceec2d16e1a None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:47.040528 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 530/1060] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:21:46 2022] GET /identity/v3/services?type=image => generated 350 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:21:47.041851 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d32f5f72-ce9d-4cfd-8ac2-27d30edfa3b4 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:47.070580 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:47.150160 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d32f5f72-ce9d-4cfd-8ac2-27d30edfa3b4 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:47.164970 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9461687b-676d-4ab6-aa36-4ceec2d16e1a None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:47.186704 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9214e3b5-8dd7-4267-aac7-ad5204ba19d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:47.187599 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9214e3b5-8dd7-4267-aac7-ad5204ba19d6 None admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:47.187946 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9214e3b5-8dd7-4267-aac7-ad5204ba19d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:47.188211 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9214e3b5-8dd7-4267-aac7-ad5204ba19d6 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:47.188875 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9214e3b5-8dd7-4267-aac7-ad5204ba19d6 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:47.190270 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9214e3b5-8dd7-4267-aac7-ad5204ba19d6 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:47.211077 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 531/1061] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:21:47 2022] POST /identity/v3/endpoints => generated 327 bytes in 170 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:21:47.279939 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9461687b-676d-4ab6-aa36-4ceec2d16e1a None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:47.324448 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70840c90-9111-4da6-9bd3-913710c7fa84 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:47.325165 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70840c90-9111-4da6-9bd3-913710c7fa84 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:47.325405 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70840c90-9111-4da6-9bd3-913710c7fa84 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:47.325634 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70840c90-9111-4da6-9bd3-913710c7fa84 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:47.338255 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70840c90-9111-4da6-9bd3-913710c7fa84 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 21:21:47.343167 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-70840c90-9111-4da6-9bd3-913710c7fa84 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 29 21:21:47.343167 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:47.343167 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:47.343167 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:47.343167 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:47.343167 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:47.343167 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:47.343167 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:47.343167 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:47.343167 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:47.343167 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:47.343167 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:47.343167 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 21:21:47.343167 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 21:21:47.343167 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 21:21:47.343167 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 21:21:47.344835 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 21:21:47.344835 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 21:21:47.344835 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 21:21:47.344835 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 21:21:47.344835 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:47.344835 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:47.344835 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:47.344835 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:47.344835 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:47.344835 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:47.344835 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:47.344835 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:47.344835 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:47.344835 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:47.344835 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:47.344835 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:47.346526 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:47.346526 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:47.346526 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:47.346526 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:47.346526 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 21:21:47.346526 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 21:21:47.346526 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 21:21:47.346526 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 21:21:47.346526 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 21:21:47.346526 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 21:21:47.346526 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 29 21:21:47.346526 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:21:47.346526 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 531/1062] 10.222.0.90 () {56 vars in 1120 bytes} [Mon Aug 29 21:21:47 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:21:47.352180 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9214e3b5-8dd7-4267-aac7-ad5204ba19d6 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:47.443981 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:47.466595 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9214e3b5-8dd7-4267-aac7-ad5204ba19d6 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:47.503096 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd7485c4-d6c4-4f8a-bc68-7b01cbd815c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:47.503709 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd7485c4-d6c4-4f8a-bc68-7b01cbd815c1 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:47.503985 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd7485c4-d6c4-4f8a-bc68-7b01cbd815c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:47.504252 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd7485c4-d6c4-4f8a-bc68-7b01cbd815c1 None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:47.504911 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd7485c4-d6c4-4f8a-bc68-7b01cbd815c1 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:47.505208 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd7485c4-d6c4-4f8a-bc68-7b01cbd815c1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:47.506629 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd7485c4-d6c4-4f8a-bc68-7b01cbd815c1 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:47.515519 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 532/1063] 10.222.0.90 () {56 vars in 1115 bytes} [Mon Aug 29 21:21:47 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:47.521569 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70840c90-9111-4da6-9bd3-913710c7fa84 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:47.550860 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:47.630220 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70840c90-9111-4da6-9bd3-913710c7fa84 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:47.666226 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28dae5c5-0312-401e-8f4b-ebafe7818df1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:47.666874 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28dae5c5-0312-401e-8f4b-ebafe7818df1 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:47.667153 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28dae5c5-0312-401e-8f4b-ebafe7818df1 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:47.667425 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28dae5c5-0312-401e-8f4b-ebafe7818df1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:47.680067 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28dae5c5-0312-401e-8f4b-ebafe7818df1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:47.681560 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28dae5c5-0312-401e-8f4b-ebafe7818df1 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:47.700234 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-28dae5c5-0312-401e-8f4b-ebafe7818df1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:21:47.700234 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:47.700234 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:47.700234 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:47.700234 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:47.700234 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:47.700234 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:47.700234 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:47.700234 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:47.700234 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:47.700234 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:47.700234 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:47.700234 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 21:21:47.700234 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 21:21:47.700234 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 21:21:47.700234 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 21:21:47.701283 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:47.701283 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:47.701283 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:47.701283 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:47.701283 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:47.701283 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:47.701283 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:47.701283 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:47.701283 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:47.701283 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:47.701283 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:47.701283 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:47.701283 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:47.701283 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:47.701283 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:47.701283 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:47.701283 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 21:21:47.702608 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 21:21:47.702608 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:21:47.702608 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:21:47.702608 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 532/1064] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:21:47 2022] GET /identity/v3/domains/Default => generated 87 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 29 21:21:47.710397 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd7485c4-d6c4-4f8a-bc68-7b01cbd815c1 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:47.816375 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd7485c4-d6c4-4f8a-bc68-7b01cbd815c1 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:47.858402 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eea9764d-aa6b-4226-967b-1b2db91d6dc3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:47.859027 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea9764d-aa6b-4226-967b-1b2db91d6dc3 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:47.859310 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea9764d-aa6b-4226-967b-1b2db91d6dc3 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:47.859893 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea9764d-aa6b-4226-967b-1b2db91d6dc3 None admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:47.860824 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eea9764d-aa6b-4226-967b-1b2db91d6dc3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:47.861125 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eea9764d-aa6b-4226-967b-1b2db91d6dc3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:47.862533 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eea9764d-aa6b-4226-967b-1b2db91d6dc3 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:47.874802 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 533/1065] 10.222.0.90 () {56 vars in 1109 bytes} [Mon Aug 29 21:21:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:21:47.881117 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28dae5c5-0312-401e-8f4b-ebafe7818df1 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:47.964411 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:47.988717 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28dae5c5-0312-401e-8f4b-ebafe7818df1 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:48.022532 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-292aba51-42ba-4936-9595-1037752a9136 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:48.023164 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-292aba51-42ba-4936-9595-1037752a9136 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:48.023446 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-292aba51-42ba-4936-9595-1037752a9136 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:48.023722 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-292aba51-42ba-4936-9595-1037752a9136 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:48.031131 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-292aba51-42ba-4936-9595-1037752a9136 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:48.032607 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-292aba51-42ba-4936-9595-1037752a9136 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:48.043723 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-292aba51-42ba-4936-9595-1037752a9136 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 29 21:21:48.043723 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:48.043723 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:48.043723 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:48.043723 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:48.043723 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:48.043723 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:48.043723 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:48.043723 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:48.043723 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:48.043723 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:48.043723 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:48.043723 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 29 21:21:48.043723 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 29 21:21:48.043723 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 29 21:21:48.043723 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 29 21:21:48.044689 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:48.044689 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:48.044689 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 29 21:21:48.044689 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 21:21:48.044689 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 29 21:21:48.044689 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 21:21:48.044689 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:48.044689 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:48.044689 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:48.044689 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:48.044689 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:48.044689 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:48.044689 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:48.044689 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:48.044689 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:48.044689 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:48.044689 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:48.044689 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:48.045661 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:48.045661 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:48.045661 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 29 21:21:48.045661 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 29 21:21:48.045661 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 29 21:21:48.045661 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 29 21:21:48.045661 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 29 21:21:48.045661 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 29 21:21:48.045661 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 29 21:21:48.045661 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:21:48.046811 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 533/1066] 10.222.0.90 () {56 vars in 1093 bytes} [Mon Aug 29 21:21:47 2022] GET /identity/v3/users/nova => generated 82 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:21:48.051581 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eea9764d-aa6b-4226-967b-1b2db91d6dc3 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:48.054529 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:48.157874 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eea9764d-aa6b-4226-967b-1b2db91d6dc3 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:48.193665 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d5bb3bb-cc15-4caa-b946-53d7ed1fe3a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:48.194305 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d5bb3bb-cc15-4caa-b946-53d7ed1fe3a0 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:48.194584 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d5bb3bb-cc15-4caa-b946-53d7ed1fe3a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:48.194852 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d5bb3bb-cc15-4caa-b946-53d7ed1fe3a0 None admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:48.195577 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d5bb3bb-cc15-4caa-b946-53d7ed1fe3a0 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:48.195882 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d5bb3bb-cc15-4caa-b946-53d7ed1fe3a0 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:48.197305 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d5bb3bb-cc15-4caa-b946-53d7ed1fe3a0 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:48.216338 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 534/1067] 10.222.0.90 () {56 vars in 1133 bytes} [Mon Aug 29 21:21:48 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:48.222538 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-292aba51-42ba-4936-9595-1037752a9136 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:48.327586 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-292aba51-42ba-4936-9595-1037752a9136 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:48.362823 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3bfce38-469b-4416-9803-d3326dd882cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:48.363454 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3bfce38-469b-4416-9803-d3326dd882cb None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:48.363730 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3bfce38-469b-4416-9803-d3326dd882cb None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:48.363998 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3bfce38-469b-4416-9803-d3326dd882cb None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:48.371039 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:48.376201 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3bfce38-469b-4416-9803-d3326dd882cb None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:48.378045 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3bfce38-469b-4416-9803-d3326dd882cb None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:48.392118 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-d3bfce38-469b-4416-9803-d3326dd882cb None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:21:48.392118 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:48.392118 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:48.392118 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:48.392118 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:48.392118 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:48.392118 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:48.392118 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:48.392118 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:48.392118 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:48.392118 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:48.392118 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:48.392118 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 21:21:48.392118 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 21:21:48.392118 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 21:21:48.392118 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 21:21:48.393163 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:48.393163 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:48.393163 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:48.393163 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:48.393163 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:48.393163 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:48.393163 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:48.393163 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:48.393163 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:48.393163 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:48.393163 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:48.393163 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:48.393163 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:48.393163 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:48.393163 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:48.393163 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:48.393163 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 21:21:48.394492 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 21:21:48.394492 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:21:48.394492 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:21:48.398072 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 534/1068] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:21:48 2022] GET /identity/v3/domains/Default => generated 87 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:21:48.399286 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d5bb3bb-cc15-4caa-b946-53d7ed1fe3a0 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:48.462018 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:48.505725 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d5bb3bb-cc15-4caa-b946-53d7ed1fe3a0 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:48.539566 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e16be7d9-90c2-4764-8f74-5aee9fed9130 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:48.540171 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e16be7d9-90c2-4764-8f74-5aee9fed9130 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:48.540449 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e16be7d9-90c2-4764-8f74-5aee9fed9130 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:48.540718 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e16be7d9-90c2-4764-8f74-5aee9fed9130 None admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:48.541350 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e16be7d9-90c2-4764-8f74-5aee9fed9130 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:48.541650 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e16be7d9-90c2-4764-8f74-5aee9fed9130 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:48.543029 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e16be7d9-90c2-4764-8f74-5aee9fed9130 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:48.559513 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 535/1069] 10.222.0.90 () {56 vars in 1109 bytes} [Mon Aug 29 21:21:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:48.561038 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3bfce38-469b-4416-9803-d3326dd882cb None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:48.561819 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-305c8e97-f733-4cf2-bca1-bc1b617a4175 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:48.562092 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-305c8e97-f733-4cf2-bca1-bc1b617a4175 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:48.562395 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-305c8e97-f733-4cf2-bca1-bc1b617a4175 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:48.564239 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 536/1070] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:21:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:21:48.569124 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff189446-3cce-4c3d-b916-1884fcd766bf None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:48.569406 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff189446-3cce-4c3d-b916-1884fcd766bf None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:48.569676 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff189446-3cce-4c3d-b916-1884fcd766bf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:48.593529 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-ff189446-3cce-4c3d-b916-1884fcd766bf None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:21:48.595668 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-ff189446-3cce-4c3d-b916-1884fcd766bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:21:48.665145 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 537/1071] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:21:48 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 29 21:21:48.671283 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3bfce38-469b-4416-9803-d3326dd882cb None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:48.708099 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09ea8e1e-aa78-4be3-a4b4-d1751aab103a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:48.708733 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ea8e1e-aa78-4be3-a4b4-d1751aab103a None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:48.709012 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ea8e1e-aa78-4be3-a4b4-d1751aab103a None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:48.709282 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ea8e1e-aa78-4be3-a4b4-d1751aab103a None admin] PATH_INFO: `/v3/projects/service` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:48.716193 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09ea8e1e-aa78-4be3-a4b4-d1751aab103a None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:48.717586 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09ea8e1e-aa78-4be3-a4b4-d1751aab103a None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:48.730165 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-09ea8e1e-aa78-4be3-a4b4-d1751aab103a None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 21:21:48.730165 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:48.730165 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:48.730165 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:48.730165 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:48.730165 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:48.730165 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:48.730165 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:48.730165 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:48.730165 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:48.730165 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:48.730165 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:48.730165 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 29 21:21:48.730165 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 29 21:21:48.730165 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 29 21:21:48.731108 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 29 21:21:48.731108 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:48.731108 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:48.731108 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:48.731108 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:48.731108 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:48.731108 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:48.731108 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:48.731108 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:48.731108 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:48.731108 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:48.731108 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:48.731108 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:48.731108 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:48.731108 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:48.731108 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:48.731108 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:48.737385 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 29 21:21:48.737385 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 29 21:21:48.737385 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 29 21:21:48.737385 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 29 21:21:48.737385 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 29 21:21:48.737385 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 29 21:21:48.737385 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 21:21:48.737385 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:21:48.737385 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 535/1072] 10.222.0.90 () {56 vars in 1111 bytes} [Mon Aug 29 21:21:48 2022] GET /identity/v3/projects/service => generated 88 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:21:48.738864 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff189446-3cce-4c3d-b916-1884fcd766bf None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:48.793051 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:48.846187 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff189446-3cce-4c3d-b916-1884fcd766bf None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:48.883294 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd9d6ea5-3b94-4aab-8525-f16652b62353 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:48.883907 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd9d6ea5-3b94-4aab-8525-f16652b62353 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:48.884186 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd9d6ea5-3b94-4aab-8525-f16652b62353 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:48.884457 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd9d6ea5-3b94-4aab-8525-f16652b62353 None admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:48.885149 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd9d6ea5-3b94-4aab-8525-f16652b62353 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:48.885513 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd9d6ea5-3b94-4aab-8525-f16652b62353 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:48.886968 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd9d6ea5-3b94-4aab-8525-f16652b62353 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:48.904199 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 538/1073] 10.222.0.90 () {56 vars in 1148 bytes} [Mon Aug 29 21:21:48 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:48.905448 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09ea8e1e-aa78-4be3-a4b4-d1751aab103a None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:48.938124 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd9d6ea5-3b94-4aab-8525-f16652b62353 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:48.959256 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:49.016583 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09ea8e1e-aa78-4be3-a4b4-d1751aab103a None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:49.054879 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28b67891-7811-4206-94dc-06250b751752 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:49.055553 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28b67891-7811-4206-94dc-06250b751752 None admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:49.055838 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28b67891-7811-4206-94dc-06250b751752 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:49.056117 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28b67891-7811-4206-94dc-06250b751752 None admin] PATH_INFO: `/v3/projects/ec03e112e3d547fbab6faf7b485c3fca/users/a35b5b7dea7b421da056f063d2436ffb/roles/9c919ca974684bc492e75274b81c78ad` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:49.056651 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd9d6ea5-3b94-4aab-8525-f16652b62353 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:49.058359 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28b67891-7811-4206-94dc-06250b751752 None admin] RBAC: Authorizing `identity:create_grant(project_id=ec03e112e3d547fbab6faf7b485c3fca, user_id=a35b5b7dea7b421da056f063d2436ffb, role_id=9c919ca974684bc492e75274b81c78ad)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:49.059788 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28b67891-7811-4206-94dc-06250b751752 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:49.069288 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:49.089887 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 536/1074] 10.222.0.90 () {58 vars in 1439 bytes} [Mon Aug 29 21:21:48 2022] PUT /identity/v3/projects/ec03e112e3d547fbab6faf7b485c3fca/users/a35b5b7dea7b421da056f063d2436ffb/roles/9c919ca974684bc492e75274b81c78ad => generated 0 bytes in 172 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 21:21:49.098867 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbfac5aa-0887-4c66-83a4-6124cc60bb63 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:49.099539 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbfac5aa-0887-4c66-83a4-6124cc60bb63 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:49.099816 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbfac5aa-0887-4c66-83a4-6124cc60bb63 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:49.100086 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbfac5aa-0887-4c66-83a4-6124cc60bb63 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:49.113976 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbfac5aa-0887-4c66-83a4-6124cc60bb63 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 21:21:49.119374 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-fbfac5aa-0887-4c66-83a4-6124cc60bb63 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 21:21:49.119374 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:49.119374 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:49.119374 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:49.119374 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:49.119374 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:49.119374 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:49.119374 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:49.119374 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:49.119374 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:49.119374 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:49.119374 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:49.119374 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 21:21:49.119374 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 21:21:49.119374 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 21:21:49.119374 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 21:21:49.120381 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 21:21:49.120381 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 21:21:49.120381 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 21:21:49.120381 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 21:21:49.120381 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:49.120381 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:49.120381 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:49.120381 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:49.120381 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:49.120381 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:49.120381 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:49.120381 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:49.120381 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:49.120381 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:49.120381 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:49.120381 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:49.121866 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:49.121866 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:49.121866 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:49.121866 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:49.121866 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 21:21:49.121866 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 21:21:49.121866 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 21:21:49.121866 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 21:21:49.121866 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 21:21:49.121866 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 21:21:49.121866 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 21:21:49.121866 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:21:49.121866 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 539/1075] 10.222.0.90 () {56 vars in 1096 bytes} [Mon Aug 29 21:21:48 2022] GET /identity/v3/roles/admin => generated 83 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 29 21:21:49.127442 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28b67891-7811-4206-94dc-06250b751752 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:49.178070 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0cb7e5e-1aa0-4996-a614-32f23af288d8 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:49.178442 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0cb7e5e-1aa0-4996-a614-32f23af288d8 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:49.178637 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0cb7e5e-1aa0-4996-a614-32f23af288d8 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:49.180002 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 540/1076] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:21:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:21:49.185512 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56d355e8-3bf6-4b1c-817e-4053437328a7 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:49.185829 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56d355e8-3bf6-4b1c-817e-4053437328a7 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:49.186108 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56d355e8-3bf6-4b1c-817e-4053437328a7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:49.205677 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-56d355e8-3bf6-4b1c-817e-4053437328a7 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:21:49.206957 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-56d355e8-3bf6-4b1c-817e-4053437328a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:21:49.246345 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28b67891-7811-4206-94dc-06250b751752 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:49.257863 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:49.285652 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 541/1077] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:21:49 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 29 21:21:49.285857 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-550ae58a-1b89-44fa-97b0-82df6882f4d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:49.286325 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-550ae58a-1b89-44fa-97b0-82df6882f4d2 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:49.286672 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-550ae58a-1b89-44fa-97b0-82df6882f4d2 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:49.286956 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-550ae58a-1b89-44fa-97b0-82df6882f4d2 None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:49.287625 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-550ae58a-1b89-44fa-97b0-82df6882f4d2 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:49.287933 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-550ae58a-1b89-44fa-97b0-82df6882f4d2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:49.289312 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-550ae58a-1b89-44fa-97b0-82df6882f4d2 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:49.299283 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 537/1078] 10.222.0.90 () {56 vars in 1099 bytes} [Mon Aug 29 21:21:49 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:49.307059 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56d355e8-3bf6-4b1c-817e-4053437328a7 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:49.405762 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:49.409308 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56d355e8-3bf6-4b1c-817e-4053437328a7 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:49.445729 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23b58a9b-8762-4cf4-90ec-2c0225ecc026 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:49.446396 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23b58a9b-8762-4cf4-90ec-2c0225ecc026 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:49.446682 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23b58a9b-8762-4cf4-90ec-2c0225ecc026 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:49.446957 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23b58a9b-8762-4cf4-90ec-2c0225ecc026 None admin] PATH_INFO: `/v3/users/f4b0f296bcd24afcb7d999777a5821e5` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:49.448113 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23b58a9b-8762-4cf4-90ec-2c0225ecc026 None admin] RBAC: Authorizing `identity:get_user(user_id=f4b0f296bcd24afcb7d999777a5821e5)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:49.449543 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23b58a9b-8762-4cf4-90ec-2c0225ecc026 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:49.456165 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 542/1079] 10.222.0.90 () {56 vars in 1177 bytes} [Mon Aug 29 21:21:49 2022] GET /identity/v3/users/f4b0f296bcd24afcb7d999777a5821e5 => generated 285 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:21:49.458218 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-550ae58a-1b89-44fa-97b0-82df6882f4d2 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:49.573979 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-550ae58a-1b89-44fa-97b0-82df6882f4d2 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:49.575241 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23b58a9b-8762-4cf4-90ec-2c0225ecc026 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:49.616736 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46615761-006d-43a0-a6bd-bf06093bb6bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:49.617383 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46615761-006d-43a0-a6bd-bf06093bb6bc None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:49.617665 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46615761-006d-43a0-a6bd-bf06093bb6bc None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:49.617973 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46615761-006d-43a0-a6bd-bf06093bb6bc None admin] PATH_INFO: `/v3/projects/4e31c4f75dea41809073912235f7ab55` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:49.619110 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46615761-006d-43a0-a6bd-bf06093bb6bc None admin] RBAC: Authorizing `identity:get_project(project_id=4e31c4f75dea41809073912235f7ab55)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:49.620628 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46615761-006d-43a0-a6bd-bf06093bb6bc None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:49.627334 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 538/1080] 10.222.0.90 () {56 vars in 1186 bytes} [Mon Aug 29 21:21:49 2022] GET /identity/v3/projects/4e31c4f75dea41809073912235f7ab55 => generated 311 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:49.628495 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46615761-006d-43a0-a6bd-bf06093bb6bc None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:49.676445 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:49.684459 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23b58a9b-8762-4cf4-90ec-2c0225ecc026 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:49.722326 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f34e7120-a126-4530-968c-e408d8f6331f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:49.722944 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f34e7120-a126-4530-968c-e408d8f6331f None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:49.723248 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f34e7120-a126-4530-968c-e408d8f6331f None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:49.723497 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f34e7120-a126-4530-968c-e408d8f6331f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:49.741428 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46615761-006d-43a0-a6bd-bf06093bb6bc None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:49.770090 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:49.780380 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1536223c-4e6d-4803-ad6d-684cdf8beb4e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:49.781047 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1536223c-4e6d-4803-ad6d-684cdf8beb4e None admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:49.781418 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1536223c-4e6d-4803-ad6d-684cdf8beb4e None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:49.782378 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1536223c-4e6d-4803-ad6d-684cdf8beb4e None admin] PATH_INFO: `/v3/projects/4e31c4f75dea41809073912235f7ab55/users/f4b0f296bcd24afcb7d999777a5821e5/roles/a6e562cdc96c4be691d638990f8403fd` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:49.784788 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1536223c-4e6d-4803-ad6d-684cdf8beb4e None admin] RBAC: Authorizing `identity:create_grant(project_id=4e31c4f75dea41809073912235f7ab55, user_id=f4b0f296bcd24afcb7d999777a5821e5, role_id=a6e562cdc96c4be691d638990f8403fd)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:49.786253 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1536223c-4e6d-4803-ad6d-684cdf8beb4e None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:49.801176 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 539/1081] 10.222.0.90 () {58 vars in 1439 bytes} [Mon Aug 29 21:21:49 2022] PUT /identity/v3/projects/4e31c4f75dea41809073912235f7ab55/users/f4b0f296bcd24afcb7d999777a5821e5/roles/a6e562cdc96c4be691d638990f8403fd => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 21:21:49.808603 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f34e7120-a126-4530-968c-e408d8f6331f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:49.810182 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f34e7120-a126-4530-968c-e408d8f6331f None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:49.959184 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 543/1082] 10.222.0.90 () {58 vars in 1282 bytes} [Mon Aug 29 21:21:49 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 384 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 29 21:21:49.965029 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1536223c-4e6d-4803-ad6d-684cdf8beb4e None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:50.069682 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:50.073662 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1536223c-4e6d-4803-ad6d-684cdf8beb4e None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:50.112359 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dae10b43-77b0-49fc-b142-3b1c11e4731e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:50.112910 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dae10b43-77b0-49fc-b142-3b1c11e4731e None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:50.113184 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dae10b43-77b0-49fc-b142-3b1c11e4731e None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:50.113451 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dae10b43-77b0-49fc-b142-3b1c11e4731e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:50.124015 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dae10b43-77b0-49fc-b142-3b1c11e4731e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:50.125492 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dae10b43-77b0-49fc-b142-3b1c11e4731e None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:50.140463 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-dae10b43-77b0-49fc-b142-3b1c11e4731e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:21:50.140463 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:50.140463 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:50.140463 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:50.140463 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:50.140463 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:50.140463 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:50.140463 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:50.140463 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:50.140463 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:50.140463 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:50.140463 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:50.140463 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 21:21:50.140463 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 21:21:50.140463 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 21:21:50.140463 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 21:21:50.141532 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:50.141532 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:50.141532 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:50.141532 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:50.141532 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:50.141532 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:50.141532 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:50.141532 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:50.141532 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:50.141532 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:50.141532 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:50.141532 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:50.141532 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:50.141532 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:50.141532 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:50.141532 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:50.141532 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 21:21:50.142537 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 21:21:50.142537 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:21:50.142537 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:21:50.142537 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 540/1083] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:21:49 2022] GET /identity/v3/domains/Default => generated 87 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:21:50.148685 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f34e7120-a126-4530-968c-e408d8f6331f None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:50.232388 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:50.270287 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f34e7120-a126-4530-968c-e408d8f6331f None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:50.304859 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24d7509b-d7da-46b3-b4d1-2ea7842e5800 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:50.305471 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d7509b-d7da-46b3-b4d1-2ea7842e5800 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:50.305753 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d7509b-d7da-46b3-b4d1-2ea7842e5800 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:50.306054 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d7509b-d7da-46b3-b4d1-2ea7842e5800 None admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:50.306698 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24d7509b-d7da-46b3-b4d1-2ea7842e5800 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:50.306998 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24d7509b-d7da-46b3-b4d1-2ea7842e5800 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:50.308355 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24d7509b-d7da-46b3-b4d1-2ea7842e5800 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:50.325831 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 544/1084] 10.222.0.90 () {56 vars in 1109 bytes} [Mon Aug 29 21:21:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:51.052205 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df303ac0-eaa8-4820-9fcc-28a14433a503 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:51.052730 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df303ac0-eaa8-4820-9fcc-28a14433a503 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:51.052730 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df303ac0-eaa8-4820-9fcc-28a14433a503 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:51.054118 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 541/1085] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:21:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:21:51.059412 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-854b64bc-f7d5-4b4e-bf12-5f49ca7f09bd None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:51.059850 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-854b64bc-f7d5-4b4e-bf12-5f49ca7f09bd None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:51.059945 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-854b64bc-f7d5-4b4e-bf12-5f49ca7f09bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:51.079335 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-854b64bc-f7d5-4b4e-bf12-5f49ca7f09bd None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:21:51.080529 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-854b64bc-f7d5-4b4e-bf12-5f49ca7f09bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:21:51.153222 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 545/1086] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:21:51 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 21:21:51.422986 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df303ac0-eaa8-4820-9fcc-28a14433a503 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:51.524795 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df303ac0-eaa8-4820-9fcc-28a14433a503 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:51.557326 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63837732-e58d-4132-ae3e-b36532b251af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:51.557976 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63837732-e58d-4132-ae3e-b36532b251af None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:51.558259 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63837732-e58d-4132-ae3e-b36532b251af None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:51.558533 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63837732-e58d-4132-ae3e-b36532b251af None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:51.569461 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63837732-e58d-4132-ae3e-b36532b251af None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 21:21:51.574269 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-63837732-e58d-4132-ae3e-b36532b251af None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 29 21:21:51.574269 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:51.574269 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:51.574269 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:51.574269 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:51.574269 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:51.574269 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:51.574269 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:51.574269 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:51.574269 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:51.574269 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:51.574269 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:51.574269 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 21:21:51.574269 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 21:21:51.574269 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 21:21:51.574269 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 21:21:51.575392 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 21:21:51.575392 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 21:21:51.575392 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 21:21:51.575392 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 21:21:51.575392 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:51.575392 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:51.575392 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:51.575392 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:51.575392 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:51.575392 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:51.575392 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:51.575392 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:51.575392 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:51.575392 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:51.575392 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:51.575392 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:51.576688 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:51.576688 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:51.576688 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:51.576688 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:51.576688 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 21:21:51.576688 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 21:21:51.576688 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 21:21:51.576688 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 21:21:51.576688 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 21:21:51.576688 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 21:21:51.576688 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 29 21:21:51.576688 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:21:51.576688 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 542/1087] 10.222.0.90 () {56 vars in 1120 bytes} [Mon Aug 29 21:21:51 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:21:51.582713 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-854b64bc-f7d5-4b4e-bf12-5f49ca7f09bd None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:51.651834 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:51.694944 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-854b64bc-f7d5-4b4e-bf12-5f49ca7f09bd None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:51.726127 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51bd0a4f-16cc-418b-ab3a-752bcc5a299b None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:51.726514 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51bd0a4f-16cc-418b-ab3a-752bcc5a299b None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:51.726879 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51bd0a4f-16cc-418b-ab3a-752bcc5a299b None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:51.728289 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 543/1088] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:21:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 29 21:21:51.730482 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5332de43-faf1-47fd-8d29-460040a1c22a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:51.731095 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5332de43-faf1-47fd-8d29-460040a1c22a None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:51.731375 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5332de43-faf1-47fd-8d29-460040a1c22a None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:51.731646 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5332de43-faf1-47fd-8d29-460040a1c22a None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:51.732313 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5332de43-faf1-47fd-8d29-460040a1c22a None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:51.732617 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5332de43-faf1-47fd-8d29-460040a1c22a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:51.733204 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4e6589a-0c62-4e6e-873e-a9d56c012d03 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:51.733489 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4e6589a-0c62-4e6e-873e-a9d56c012d03 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:51.733764 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4e6589a-0c62-4e6e-873e-a9d56c012d03 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:51.734007 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5332de43-faf1-47fd-8d29-460040a1c22a None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:51.743263 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 546/1089] 10.222.0.90 () {56 vars in 1115 bytes} [Mon Aug 29 21:21:51 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:51.748896 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5332de43-faf1-47fd-8d29-460040a1c22a None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:51.751777 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-e4e6589a-0c62-4e6e-873e-a9d56c012d03 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:21:51.752983 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-e4e6589a-0c62-4e6e-873e-a9d56c012d03 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:21:51.760758 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:51.822072 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 544/1090] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:21:51 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 29 21:21:51.862615 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5332de43-faf1-47fd-8d29-460040a1c22a None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:51.882639 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:51.898819 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdd00401-d678-421a-b598-07f3d1a871a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:51.899457 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd00401-d678-421a-b598-07f3d1a871a8 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:51.899740 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd00401-d678-421a-b598-07f3d1a871a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:51.900016 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd00401-d678-421a-b598-07f3d1a871a8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:51.912974 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdd00401-d678-421a-b598-07f3d1a871a8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:51.914489 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdd00401-d678-421a-b598-07f3d1a871a8 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:51.929136 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-fdd00401-d678-421a-b598-07f3d1a871a8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:21:51.929136 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:51.929136 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:51.929136 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:51.929136 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:51.929136 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:51.929136 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:51.929136 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:51.929136 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:51.929136 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:51.929136 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:51.929136 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:51.929136 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 21:21:51.929136 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 21:21:51.929136 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 21:21:51.929136 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 21:21:51.930239 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:51.930239 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:51.930239 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:51.930239 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:51.930239 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:51.930239 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:51.930239 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:51.930239 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:51.930239 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:51.930239 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:51.930239 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:51.930239 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:51.930239 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:51.930239 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:51.930239 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:51.930239 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:51.930239 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 21:21:51.931163 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 21:21:51.931163 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:21:51.931163 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:21:51.931163 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 547/1091] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:21:51 2022] GET /identity/v3/domains/Default => generated 87 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 29 21:21:51.936606 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4e6589a-0c62-4e6e-873e-a9d56c012d03 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:52.035512 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4e6589a-0c62-4e6e-873e-a9d56c012d03 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:52.069660 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1defb704-6f4e-48b1-bbf3-d75fc02c82f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:52.070322 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1defb704-6f4e-48b1-bbf3-d75fc02c82f7 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:52.070570 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1defb704-6f4e-48b1-bbf3-d75fc02c82f7 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:52.070846 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1defb704-6f4e-48b1-bbf3-d75fc02c82f7 None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:52.071491 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1defb704-6f4e-48b1-bbf3-d75fc02c82f7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:52.071796 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1defb704-6f4e-48b1-bbf3-d75fc02c82f7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:52.073166 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1defb704-6f4e-48b1-bbf3-d75fc02c82f7 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:52.084856 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 545/1092] 10.222.0.90 () {56 vars in 1109 bytes} [Mon Aug 29 21:21:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:52.090331 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdd00401-d678-421a-b598-07f3d1a871a8 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:52.097672 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1defb704-6f4e-48b1-bbf3-d75fc02c82f7 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:52.100426 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:52.194253 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdd00401-d678-421a-b598-07f3d1a871a8 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:52.202400 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1defb704-6f4e-48b1-bbf3-d75fc02c82f7 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:52.228382 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c465d07-fd6f-40d7-adaa-773f197c4ea6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:52.229006 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c465d07-fd6f-40d7-adaa-773f197c4ea6 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:52.229278 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c465d07-fd6f-40d7-adaa-773f197c4ea6 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:52.229552 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c465d07-fd6f-40d7-adaa-773f197c4ea6 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:52.236912 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d970853-0c58-4072-9100-4085919db749 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:52.237441 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c465d07-fd6f-40d7-adaa-773f197c4ea6 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:52.237563 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d970853-0c58-4072-9100-4085919db749 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:52.237808 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d970853-0c58-4072-9100-4085919db749 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:52.238102 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d970853-0c58-4072-9100-4085919db749 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:52.238952 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c465d07-fd6f-40d7-adaa-773f197c4ea6 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:52.248388 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d970853-0c58-4072-9100-4085919db749 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 21:21:52.251104 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-1c465d07-fd6f-40d7-adaa-773f197c4ea6 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 29 21:21:52.251104 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:52.251104 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:52.251104 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:52.251104 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:52.251104 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:52.251104 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:52.251104 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:52.251104 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:52.251104 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:52.251104 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:52.251104 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:52.251104 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 29 21:21:52.251104 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 29 21:21:52.251104 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 29 21:21:52.251104 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 29 21:21:52.252054 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:52.252054 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:52.252054 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 29 21:21:52.252054 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 21:21:52.252054 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 29 21:21:52.252054 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 21:21:52.252054 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:52.252054 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:52.252054 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:52.252054 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:52.252054 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:52.252054 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:52.252054 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:52.252054 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:52.252054 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:52.252054 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:52.252054 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:52.252054 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:52.253181 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:52.253181 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:52.253181 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 29 21:21:52.253181 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 29 21:21:52.253181 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 29 21:21:52.253181 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 29 21:21:52.253181 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 29 21:21:52.253181 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 29 21:21:52.253181 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 29 21:21:52.253181 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:21:52.253181 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 548/1093] 10.222.0.90 () {56 vars in 1093 bytes} [Mon Aug 29 21:21:52 2022] GET /identity/v3/users/nova => generated 82 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:21:52.254130 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-7d970853-0c58-4072-9100-4085919db749 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 21:21:52.254130 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:52.254130 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:52.254130 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:52.254130 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:52.254130 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:52.254130 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:52.254130 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:52.254130 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:52.254130 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:52.254130 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:52.254130 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:52.254130 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 21:21:52.254130 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 21:21:52.254130 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 21:21:52.254130 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 21:21:52.255180 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 21:21:52.255180 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 21:21:52.255180 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 21:21:52.255180 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 21:21:52.255180 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:52.255180 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:52.255180 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:52.255180 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:52.255180 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:52.255180 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:52.255180 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:52.255180 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:52.255180 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:52.255180 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:52.255180 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:52.255180 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:52.256293 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:52.256293 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:52.256293 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:52.256293 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:52.256293 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 21:21:52.256293 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 21:21:52.256293 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 21:21:52.256293 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 21:21:52.256293 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 21:21:52.256293 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 21:21:52.256293 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 21:21:52.256293 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:21:52.258078 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 546/1094] 10.222.0.90 () {56 vars in 1096 bytes} [Mon Aug 29 21:21:52 2022] GET /identity/v3/roles/admin => generated 83 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 29 21:21:52.258184 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c465d07-fd6f-40d7-adaa-773f197c4ea6 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:52.260531 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d970853-0c58-4072-9100-4085919db749 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:52.290373 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:52.321833 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:52.360694 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c465d07-fd6f-40d7-adaa-773f197c4ea6 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:52.362373 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d970853-0c58-4072-9100-4085919db749 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:52.394419 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf3db6e4-e91b-4150-ad5e-c86e6f8f0afb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:52.395030 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf3db6e4-e91b-4150-ad5e-c86e6f8f0afb None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:52.395318 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf3db6e4-e91b-4150-ad5e-c86e6f8f0afb None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:52.395591 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf3db6e4-e91b-4150-ad5e-c86e6f8f0afb None admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:52.396316 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf3db6e4-e91b-4150-ad5e-c86e6f8f0afb None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:52.396625 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf3db6e4-e91b-4150-ad5e-c86e6f8f0afb None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:52.397317 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad7a2172-5c4e-4ca6-a66e-1ac584b4fe0b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:52.397966 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad7a2172-5c4e-4ca6-a66e-1ac584b4fe0b None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:52.398063 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf3db6e4-e91b-4150-ad5e-c86e6f8f0afb None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:52.398254 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad7a2172-5c4e-4ca6-a66e-1ac584b4fe0b None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:52.398529 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad7a2172-5c4e-4ca6-a66e-1ac584b4fe0b None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:52.399187 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad7a2172-5c4e-4ca6-a66e-1ac584b4fe0b None admin] RBAC: Adding query filter params (name=admin) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:52.399604 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad7a2172-5c4e-4ca6-a66e-1ac584b4fe0b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:52.400839 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad7a2172-5c4e-4ca6-a66e-1ac584b4fe0b None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:52.409751 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 547/1095] 10.222.0.90 () {56 vars in 1099 bytes} [Mon Aug 29 21:21:52 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:21:52.419764 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 549/1096] 10.222.0.90 () {56 vars in 1133 bytes} [Mon Aug 29 21:21:52 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:21:52.419931 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad7a2172-5c4e-4ca6-a66e-1ac584b4fe0b None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:52.420773 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf3db6e4-e91b-4150-ad5e-c86e6f8f0afb None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:52.511741 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:52.516541 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad7a2172-5c4e-4ca6-a66e-1ac584b4fe0b None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:52.521935 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf3db6e4-e91b-4150-ad5e-c86e6f8f0afb None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:52.542911 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:52.554693 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dc6b1a8-d0ca-484e-8229-fb36e67a19f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:52.555329 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc6b1a8-d0ca-484e-8229-fb36e67a19f7 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:52.555601 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc6b1a8-d0ca-484e-8229-fb36e67a19f7 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:52.555872 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc6b1a8-d0ca-484e-8229-fb36e67a19f7 None admin] PATH_INFO: `/v3/users/f4b0f296bcd24afcb7d999777a5821e5` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:52.556888 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b14fca1-c6b3-4284-87b0-198719caaf57 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:52.557020 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dc6b1a8-d0ca-484e-8229-fb36e67a19f7 None admin] RBAC: Authorizing `identity:get_user(user_id=f4b0f296bcd24afcb7d999777a5821e5)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:52.557495 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b14fca1-c6b3-4284-87b0-198719caaf57 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:52.557773 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b14fca1-c6b3-4284-87b0-198719caaf57 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:52.558074 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b14fca1-c6b3-4284-87b0-198719caaf57 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:52.558419 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dc6b1a8-d0ca-484e-8229-fb36e67a19f7 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:52.560065 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 548/1097] 10.222.0.90 () {56 vars in 1177 bytes} [Mon Aug 29 21:21:52 2022] GET /identity/v3/users/f4b0f296bcd24afcb7d999777a5821e5 => generated 285 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:52.566804 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dc6b1a8-d0ca-484e-8229-fb36e67a19f7 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:52.628615 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b14fca1-c6b3-4284-87b0-198719caaf57 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:52.630216 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b14fca1-c6b3-4284-87b0-198719caaf57 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:52.665288 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dc6b1a8-d0ca-484e-8229-fb36e67a19f7 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:52.700724 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8774ef63-5919-4794-9e97-f5189fd44c12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:52.701331 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8774ef63-5919-4794-9e97-f5189fd44c12 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:52.701610 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8774ef63-5919-4794-9e97-f5189fd44c12 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:52.701921 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8774ef63-5919-4794-9e97-f5189fd44c12 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:52.717192 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:52.728521 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 550/1098] 10.222.0.90 () {58 vars in 1282 bytes} [Mon Aug 29 21:21:52 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 308 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Aug 29 21:21:52.734987 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b14fca1-c6b3-4284-87b0-198719caaf57 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:52.761955 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:52.772151 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8774ef63-5919-4794-9e97-f5189fd44c12 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:52.773627 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8774ef63-5919-4794-9e97-f5189fd44c12 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:52.833771 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b14fca1-c6b3-4284-87b0-198719caaf57 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:52.873942 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81784ba1-6259-4e5e-bf10-40da50e2795d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:52.874609 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81784ba1-6259-4e5e-bf10-40da50e2795d None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:52.874914 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81784ba1-6259-4e5e-bf10-40da50e2795d None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:52.875305 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81784ba1-6259-4e5e-bf10-40da50e2795d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:52.879732 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 549/1099] 10.222.0.90 () {58 vars in 1282 bytes} [Mon Aug 29 21:21:52 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 313 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Aug 29 21:21:52.885511 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8774ef63-5919-4794-9e97-f5189fd44c12 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:52.886665 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81784ba1-6259-4e5e-bf10-40da50e2795d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:52.888116 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81784ba1-6259-4e5e-bf10-40da50e2795d None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:52.902707 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-81784ba1-6259-4e5e-bf10-40da50e2795d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:21:52.902707 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:52.902707 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:52.902707 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:52.902707 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:52.902707 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:52.902707 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:52.902707 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:52.902707 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:52.902707 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:52.902707 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:52.902707 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:52.902707 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 21:21:52.902707 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 21:21:52.902707 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 21:21:52.902707 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 21:21:52.903715 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:52.903715 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:52.903715 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:52.903715 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:52.903715 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:52.903715 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:52.903715 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:52.903715 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:52.903715 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:52.903715 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:52.903715 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:52.903715 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:52.903715 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:52.903715 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:52.903715 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:52.903715 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:52.903715 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 21:21:52.904654 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 21:21:52.904654 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:21:52.904654 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:21:52.904654 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 551/1100] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:21:52 2022] GET /identity/v3/domains/Default => generated 87 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 29 21:21:52.910197 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81784ba1-6259-4e5e-bf10-40da50e2795d None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:52.952648 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:52.975058 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:52.985050 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8774ef63-5919-4794-9e97-f5189fd44c12 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:53.014189 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81784ba1-6259-4e5e-bf10-40da50e2795d None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:53.019490 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b13ce102-9e94-4ddb-a630-99e9c2b3d57b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:53.020124 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b13ce102-9e94-4ddb-a630-99e9c2b3d57b None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:53.020405 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b13ce102-9e94-4ddb-a630-99e9c2b3d57b None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:53.020681 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b13ce102-9e94-4ddb-a630-99e9c2b3d57b None admin] PATH_INFO: `/v3/projects/4e31c4f75dea41809073912235f7ab55` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:53.021828 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b13ce102-9e94-4ddb-a630-99e9c2b3d57b None admin] RBAC: Authorizing `identity:get_project(project_id=4e31c4f75dea41809073912235f7ab55)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:53.023271 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b13ce102-9e94-4ddb-a630-99e9c2b3d57b None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:53.024900 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 550/1101] 10.222.0.90 () {56 vars in 1186 bytes} [Mon Aug 29 21:21:52 2022] GET /identity/v3/projects/4e31c4f75dea41809073912235f7ab55 => generated 311 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:21:53.030850 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b13ce102-9e94-4ddb-a630-99e9c2b3d57b None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:53.048560 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0c93e0e-5c0c-4b30-a62d-22828c9ea0ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:53.049172 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0c93e0e-5c0c-4b30-a62d-22828c9ea0ec None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:53.049454 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0c93e0e-5c0c-4b30-a62d-22828c9ea0ec None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:53.049725 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0c93e0e-5c0c-4b30-a62d-22828c9ea0ec None admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:53.050403 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0c93e0e-5c0c-4b30-a62d-22828c9ea0ec None admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:53.050707 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0c93e0e-5c0c-4b30-a62d-22828c9ea0ec None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:53.052071 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0c93e0e-5c0c-4b30-a62d-22828c9ea0ec None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:53.063808 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 552/1102] 10.222.0.90 () {56 vars in 1109 bytes} [Mon Aug 29 21:21:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:21:53.069361 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0c93e0e-5c0c-4b30-a62d-22828c9ea0ec None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:53.132770 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b13ce102-9e94-4ddb-a630-99e9c2b3d57b None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:53.167623 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab38e1b6-8584-46eb-9526-689ecdb87f91 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:53.168267 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab38e1b6-8584-46eb-9526-689ecdb87f91 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:53.168536 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab38e1b6-8584-46eb-9526-689ecdb87f91 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:53.168820 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab38e1b6-8584-46eb-9526-689ecdb87f91 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:53.169518 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab38e1b6-8584-46eb-9526-689ecdb87f91 None admin] RBAC: Adding query filter params (role.id=a6e562cdc96c4be691d638990f8403fd, scope.project.id=4e31c4f75dea41809073912235f7ab55, user.id=f4b0f296bcd24afcb7d999777a5821e5) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:53.169855 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab38e1b6-8584-46eb-9526-689ecdb87f91 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:53.171283 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab38e1b6-8584-46eb-9526-689ecdb87f91 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:53.172863 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0c93e0e-5c0c-4b30-a62d-22828c9ea0ec None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:53.174761 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:53.178228 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 551/1103] 10.222.0.90 () {56 vars in 1374 bytes} [Mon Aug 29 21:21:53 2022] GET /identity/v3/role_assignments?user.id=f4b0f296bcd24afcb7d999777a5821e5&scope.project.id=4e31c4f75dea41809073912235f7ab55&role.id=a6e562cdc96c4be691d638990f8403fd => generated 608 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:21:53.212884 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2b5e9ce-95fd-4900-b639-6e77ea0b79a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:53.213539 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2b5e9ce-95fd-4900-b639-6e77ea0b79a1 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:53.213840 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2b5e9ce-95fd-4900-b639-6e77ea0b79a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:53.214235 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2b5e9ce-95fd-4900-b639-6e77ea0b79a1 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:53.222945 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2b5e9ce-95fd-4900-b639-6e77ea0b79a1 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:53.224448 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2b5e9ce-95fd-4900-b639-6e77ea0b79a1 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:53.237361 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-e2b5e9ce-95fd-4900-b639-6e77ea0b79a1 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 21:21:53.237361 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:21:53.237361 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:21:53.237361 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:21:53.237361 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:21:53.237361 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:21:53.237361 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:21:53.237361 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:21:53.237361 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:21:53.237361 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:21:53.237361 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:21:53.237361 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:21:53.237361 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 29 21:21:53.237361 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 29 21:21:53.237361 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 29 21:21:53.238375 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 29 21:21:53.238375 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:21:53.238375 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:21:53.238375 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:21:53.238375 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:21:53.238375 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:21:53.238375 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:21:53.238375 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:21:53.238375 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:21:53.238375 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:21:53.238375 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:21:53.238375 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:21:53.238375 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:21:53.238375 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:21:53.238375 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:21:53.238375 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:21:53.238375 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:21:53.239298 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 29 21:21:53.239298 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 29 21:21:53.239298 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 29 21:21:53.239298 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 29 21:21:53.239298 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 29 21:21:53.239298 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 29 21:21:53.239298 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 21:21:53.239298 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:21:53.239298 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 553/1104] 10.222.0.90 () {56 vars in 1111 bytes} [Mon Aug 29 21:21:53 2022] GET /identity/v3/projects/service => generated 88 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 29 21:21:53.244956 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab38e1b6-8584-46eb-9526-689ecdb87f91 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:53.272840 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:53.351945 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab38e1b6-8584-46eb-9526-689ecdb87f91 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:53.385358 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab234431-ed34-445a-b5ac-7b8b4ea148a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:53.385931 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab234431-ed34-445a-b5ac-7b8b4ea148a4 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:53.386216 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab234431-ed34-445a-b5ac-7b8b4ea148a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:53.386492 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab234431-ed34-445a-b5ac-7b8b4ea148a4 None admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:53.387184 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab234431-ed34-445a-b5ac-7b8b4ea148a4 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:53.387494 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab234431-ed34-445a-b5ac-7b8b4ea148a4 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:53.388922 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab234431-ed34-445a-b5ac-7b8b4ea148a4 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:53.401476 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 552/1105] 10.222.0.90 () {56 vars in 1148 bytes} [Mon Aug 29 21:21:53 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:53.408392 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2b5e9ce-95fd-4900-b639-6e77ea0b79a1 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:53.510623 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2b5e9ce-95fd-4900-b639-6e77ea0b79a1 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:53.544772 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41c1f251-022e-4ead-8063-cdcab5cbfb65 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:53.545382 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41c1f251-022e-4ead-8063-cdcab5cbfb65 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:53.545666 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41c1f251-022e-4ead-8063-cdcab5cbfb65 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:53.545967 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41c1f251-022e-4ead-8063-cdcab5cbfb65 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:53.546909 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41c1f251-022e-4ead-8063-cdcab5cbfb65 None admin] RBAC: Adding query filter params (role.id=9c919ca974684bc492e75274b81c78ad, scope.project.id=ec03e112e3d547fbab6faf7b485c3fca, user.id=a35b5b7dea7b421da056f063d2436ffb) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:53.547332 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41c1f251-022e-4ead-8063-cdcab5cbfb65 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:53.548767 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41c1f251-022e-4ead-8063-cdcab5cbfb65 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:53.554702 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 554/1106] 10.222.0.90 () {56 vars in 1374 bytes} [Mon Aug 29 21:21:53 2022] GET /identity/v3/role_assignments?user.id=a35b5b7dea7b421da056f063d2436ffb&scope.project.id=ec03e112e3d547fbab6faf7b485c3fca&role.id=9c919ca974684bc492e75274b81c78ad => generated 608 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:55.124504 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d85fb145-166a-41e1-aaeb-98d8a4c3d374 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:55.125516 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d85fb145-166a-41e1-aaeb-98d8a4c3d374 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:55.125516 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d85fb145-166a-41e1-aaeb-98d8a4c3d374 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:55.127046 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 553/1107] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:21:55 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:21:55.132360 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e52f43c-9755-4ccd-bff9-035364e1f44d None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:55.132784 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e52f43c-9755-4ccd-bff9-035364e1f44d None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:55.132917 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e52f43c-9755-4ccd-bff9-035364e1f44d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:55.151570 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-5e52f43c-9755-4ccd-bff9-035364e1f44d None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:21:55.152754 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-5e52f43c-9755-4ccd-bff9-035364e1f44d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:21:55.172233 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:55.219837 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 555/1108] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:21:55 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 21:21:55.487964 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d85fb145-166a-41e1-aaeb-98d8a4c3d374 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:55.572348 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:55.589362 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d85fb145-166a-41e1-aaeb-98d8a4c3d374 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:55.622495 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37e14972-2f1a-4881-aaa1-bd847eb0a8f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:55.623110 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37e14972-2f1a-4881-aaa1-bd847eb0a8f1 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:55.623388 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37e14972-2f1a-4881-aaa1-bd847eb0a8f1 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:55.623659 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37e14972-2f1a-4881-aaa1-bd847eb0a8f1 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:55.624837 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37e14972-2f1a-4881-aaa1-bd847eb0a8f1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:55.626794 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37e14972-2f1a-4881-aaa1-bd847eb0a8f1 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:55.628334 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 554/1109] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:21:55 2022] GET /identity/v3/domains/default => generated 199 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:55.635926 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e52f43c-9755-4ccd-bff9-035364e1f44d None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:55.736778 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e52f43c-9755-4ccd-bff9-035364e1f44d None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:55.770392 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfe963cb-d568-46e6-b146-a3f49104cb8f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:55.770933 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfe963cb-d568-46e6-b146-a3f49104cb8f None admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:55.771211 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfe963cb-d568-46e6-b146-a3f49104cb8f None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:55.771480 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfe963cb-d568-46e6-b146-a3f49104cb8f None admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:55.772140 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfe963cb-d568-46e6-b146-a3f49104cb8f None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:55.773558 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfe963cb-d568-46e6-b146-a3f49104cb8f None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:55.794487 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 556/1110] 10.222.0.90 () {60 vars in 1132 bytes} [Mon Aug 29 21:21:55 2022] POST /identity/v3/users => generated 286 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:21:57.747333 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-593e95be-031f-45ac-88a0-aefc29cfb14c None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:57.747903 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-593e95be-031f-45ac-88a0-aefc29cfb14c None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:57.747903 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-593e95be-031f-45ac-88a0-aefc29cfb14c None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:57.749558 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 555/1111] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:21:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:21:57.754792 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c840a2f-347d-4259-b0e6-f13063e06b21 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:57.755225 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c840a2f-347d-4259-b0e6-f13063e06b21 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:57.755353 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c840a2f-347d-4259-b0e6-f13063e06b21 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:57.773747 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-0c840a2f-347d-4259-b0e6-f13063e06b21 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:21:57.774995 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-0c840a2f-347d-4259-b0e6-f13063e06b21 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:21:57.782825 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:57.844676 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 557/1112] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:21:57 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 21:21:58.114690 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-593e95be-031f-45ac-88a0-aefc29cfb14c None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:58.214780 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-593e95be-031f-45ac-88a0-aefc29cfb14c None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:58.248276 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85ec3b1e-1775-4648-b1b5-a8fa3a181c33 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:58.248895 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ec3b1e-1775-4648-b1b5-a8fa3a181c33 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:58.249174 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ec3b1e-1775-4648-b1b5-a8fa3a181c33 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:58.249446 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ec3b1e-1775-4648-b1b5-a8fa3a181c33 None admin] PATH_INFO: `/v3/roles/9745586777bf473bab7d887bf25bf236` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:58.251652 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85ec3b1e-1775-4648-b1b5-a8fa3a181c33 None admin] RBAC: Authorizing `identity:get_role(role_id=9745586777bf473bab7d887bf25bf236)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:58.253014 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85ec3b1e-1775-4648-b1b5-a8fa3a181c33 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:58.254357 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 556/1113] 10.222.0.90 () {56 vars in 1177 bytes} [Mon Aug 29 21:21:58 2022] GET /identity/v3/roles/9745586777bf473bab7d887bf25bf236 => generated 223 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:58.260149 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c840a2f-347d-4259-b0e6-f13063e06b21 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:58.356576 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c840a2f-347d-4259-b0e6-f13063e06b21 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:58.390060 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0fe92c3-c6a5-4ba4-8ecc-02c6acb0084c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:58.391544 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0fe92c3-c6a5-4ba4-8ecc-02c6acb0084c None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:58.391829 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0fe92c3-c6a5-4ba4-8ecc-02c6acb0084c None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:58.392103 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0fe92c3-c6a5-4ba4-8ecc-02c6acb0084c None admin] PATH_INFO: `/v3/users/86790606a1a848088b14ac17e2a21d71` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:58.402844 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:58.405425 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0fe92c3-c6a5-4ba4-8ecc-02c6acb0084c None admin] RBAC: Authorizing `identity:get_user(user_id=86790606a1a848088b14ac17e2a21d71)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:58.407260 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0fe92c3-c6a5-4ba4-8ecc-02c6acb0084c None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:58.409093 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 558/1114] 10.222.0.90 () {56 vars in 1177 bytes} [Mon Aug 29 21:21:58 2022] GET /identity/v3/users/86790606a1a848088b14ac17e2a21d71 => generated 286 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:58.415027 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85ec3b1e-1775-4648-b1b5-a8fa3a181c33 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:58.436445 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:58.521666 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85ec3b1e-1775-4648-b1b5-a8fa3a181c33 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:58.554890 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b8be17c-f919-45a7-a281-e7c9b9e5c79f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:58.555492 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b8be17c-f919-45a7-a281-e7c9b9e5c79f None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:58.555772 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b8be17c-f919-45a7-a281-e7c9b9e5c79f None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:58.556045 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b8be17c-f919-45a7-a281-e7c9b9e5c79f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:58.621756 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b8be17c-f919-45a7-a281-e7c9b9e5c79f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:58.623226 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b8be17c-f919-45a7-a281-e7c9b9e5c79f None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:58.643642 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:58.725544 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 557/1115] 10.222.0.90 () {58 vars in 1282 bytes} [Mon Aug 29 21:21:58 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 29 21:21:58.731469 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0fe92c3-c6a5-4ba4-8ecc-02c6acb0084c None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:58.831447 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0fe92c3-c6a5-4ba4-8ecc-02c6acb0084c None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:58.866149 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2abb6435-570c-4044-a0a9-a662992d877a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:58.866779 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2abb6435-570c-4044-a0a9-a662992d877a None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:58.867058 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2abb6435-570c-4044-a0a9-a662992d877a None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:58.867330 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2abb6435-570c-4044-a0a9-a662992d877a None admin] PATH_INFO: `/v3/projects/4e31c4f75dea41809073912235f7ab55` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:58.868428 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2abb6435-570c-4044-a0a9-a662992d877a None admin] RBAC: Authorizing `identity:get_project(project_id=4e31c4f75dea41809073912235f7ab55)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:58.869888 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2abb6435-570c-4044-a0a9-a662992d877a None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:58.871504 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 559/1116] 10.222.0.90 () {56 vars in 1186 bytes} [Mon Aug 29 21:21:58 2022] GET /identity/v3/projects/4e31c4f75dea41809073912235f7ab55 => generated 311 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:21:58.877742 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b8be17c-f919-45a7-a281-e7c9b9e5c79f None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:21:58.975875 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b8be17c-f919-45a7-a281-e7c9b9e5c79f None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:21:58.996077 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:21:59.010586 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97dfdd67-5f61-4fd6-bfa6-63edcb8e6df9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:21:59.011184 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97dfdd67-5f61-4fd6-bfa6-63edcb8e6df9 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:21:59.011461 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97dfdd67-5f61-4fd6-bfa6-63edcb8e6df9 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:21:59.011733 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97dfdd67-5f61-4fd6-bfa6-63edcb8e6df9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:21:59.012449 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97dfdd67-5f61-4fd6-bfa6-63edcb8e6df9 None admin] RBAC: Adding query filter params (role.id=9745586777bf473bab7d887bf25bf236, scope.project.id=4e31c4f75dea41809073912235f7ab55, user.id=86790606a1a848088b14ac17e2a21d71) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:21:59.012756 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97dfdd67-5f61-4fd6-bfa6-63edcb8e6df9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:21:59.014209 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97dfdd67-5f61-4fd6-bfa6-63edcb8e6df9 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:21:59.019985 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 558/1117] 10.222.0.90 () {56 vars in 1374 bytes} [Mon Aug 29 21:21:58 2022] GET /identity/v3/role_assignments?user.id=86790606a1a848088b14ac17e2a21d71&scope.project.id=4e31c4f75dea41809073912235f7ab55&role.id=9745586777bf473bab7d887bf25bf236 => generated 259 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:22:00.956983 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9c9d7ad-bc7f-4e50-8dfc-8c9665eb5b8e None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:22:00.957557 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9c9d7ad-bc7f-4e50-8dfc-8c9665eb5b8e None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:22:00.957557 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9c9d7ad-bc7f-4e50-8dfc-8c9665eb5b8e None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:22:00.959333 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 560/1118] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:22:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:22:00.964614 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba7bd3b-2ea3-497d-9e23-2c1197c3ab94 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:22:00.965011 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba7bd3b-2ea3-497d-9e23-2c1197c3ab94 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:22:00.965163 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba7bd3b-2ea3-497d-9e23-2c1197c3ab94 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:22:00.983351 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-0ba7bd3b-2ea3-497d-9e23-2c1197c3ab94 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:22:00.985059 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-0ba7bd3b-2ea3-497d-9e23-2c1197c3ab94 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:22:01.050869 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 559/1119] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:22:00 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 21:22:01.317952 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9c9d7ad-bc7f-4e50-8dfc-8c9665eb5b8e None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:22:01.385054 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:22:01.418967 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9c9d7ad-bc7f-4e50-8dfc-8c9665eb5b8e None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:22:01.452518 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcf2f21e-797f-41c0-b53e-c774c2893838 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:22:01.453143 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf2f21e-797f-41c0-b53e-c774c2893838 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:22:01.453450 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf2f21e-797f-41c0-b53e-c774c2893838 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:22:01.453729 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf2f21e-797f-41c0-b53e-c774c2893838 None admin] PATH_INFO: `/v3/roles/9745586777bf473bab7d887bf25bf236` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:22:01.455163 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcf2f21e-797f-41c0-b53e-c774c2893838 None admin] RBAC: Authorizing `identity:get_role(role_id=9745586777bf473bab7d887bf25bf236)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:22:01.456525 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcf2f21e-797f-41c0-b53e-c774c2893838 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:22:01.457869 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 561/1120] 10.222.0.90 () {56 vars in 1177 bytes} [Mon Aug 29 21:22:01 2022] GET /identity/v3/roles/9745586777bf473bab7d887bf25bf236 => generated 223 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:22:01.463666 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ba7bd3b-2ea3-497d-9e23-2c1197c3ab94 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:22:01.558826 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ba7bd3b-2ea3-497d-9e23-2c1197c3ab94 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:22:01.572950 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:22:01.592455 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7305b8bd-2ac8-4101-ba62-116ae6592466 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:22:01.593075 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7305b8bd-2ac8-4101-ba62-116ae6592466 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:22:01.593353 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7305b8bd-2ac8-4101-ba62-116ae6592466 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:22:01.593623 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7305b8bd-2ac8-4101-ba62-116ae6592466 None admin] PATH_INFO: `/v3/users/86790606a1a848088b14ac17e2a21d71` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:22:01.594952 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7305b8bd-2ac8-4101-ba62-116ae6592466 None admin] RBAC: Authorizing `identity:get_user(user_id=86790606a1a848088b14ac17e2a21d71)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:22:01.596648 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7305b8bd-2ac8-4101-ba62-116ae6592466 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:22:01.598242 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 560/1121] 10.222.0.90 () {56 vars in 1177 bytes} [Mon Aug 29 21:22:01 2022] GET /identity/v3/users/86790606a1a848088b14ac17e2a21d71 => generated 286 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:22:01.603943 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcf2f21e-797f-41c0-b53e-c774c2893838 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:22:01.706620 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcf2f21e-797f-41c0-b53e-c774c2893838 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:22:01.740301 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e0a3fda-cc61-414e-85f0-b7f29e50ea70 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:22:01.740922 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e0a3fda-cc61-414e-85f0-b7f29e50ea70 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:22:01.741203 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e0a3fda-cc61-414e-85f0-b7f29e50ea70 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:22:01.741475 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e0a3fda-cc61-414e-85f0-b7f29e50ea70 None admin] PATH_INFO: `/v3/projects/4e31c4f75dea41809073912235f7ab55` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:22:01.742619 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e0a3fda-cc61-414e-85f0-b7f29e50ea70 None admin] RBAC: Authorizing `identity:get_project(project_id=4e31c4f75dea41809073912235f7ab55)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:22:01.744041 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e0a3fda-cc61-414e-85f0-b7f29e50ea70 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:22:01.745664 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 562/1122] 10.222.0.90 () {56 vars in 1186 bytes} [Mon Aug 29 21:22:01 2022] GET /identity/v3/projects/4e31c4f75dea41809073912235f7ab55 => generated 311 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:22:01.752848 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7305b8bd-2ac8-4101-ba62-116ae6592466 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:22:01.852765 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7305b8bd-2ac8-4101-ba62-116ae6592466 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:22:01.895698 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91308194-5391-4673-95b6-d36e972288b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:22:01.896363 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91308194-5391-4673-95b6-d36e972288b2 None admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:22:01.896642 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91308194-5391-4673-95b6-d36e972288b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:22:01.896915 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91308194-5391-4673-95b6-d36e972288b2 None admin] PATH_INFO: `/v3/projects/4e31c4f75dea41809073912235f7ab55/users/86790606a1a848088b14ac17e2a21d71/roles/9745586777bf473bab7d887bf25bf236` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:22:01.899074 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91308194-5391-4673-95b6-d36e972288b2 None admin] RBAC: Authorizing `identity:create_grant(project_id=4e31c4f75dea41809073912235f7ab55, user_id=86790606a1a848088b14ac17e2a21d71, role_id=9745586777bf473bab7d887bf25bf236)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:22:01.900507 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91308194-5391-4673-95b6-d36e972288b2 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:22:01.913286 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 561/1123] 10.222.0.90 () {58 vars in 1439 bytes} [Mon Aug 29 21:22:01 2022] PUT /identity/v3/projects/4e31c4f75dea41809073912235f7ab55/users/86790606a1a848088b14ac17e2a21d71/roles/9745586777bf473bab7d887bf25bf236 => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:22:03.795793 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff061057-5023-4425-9c41-851e9a20311c None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:22:03.796325 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff061057-5023-4425-9c41-851e9a20311c None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:22:03.796325 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff061057-5023-4425-9c41-851e9a20311c None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:22:03.798244 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 563/1124] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:22:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:22:03.803238 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cedf5c7-cde6-47b7-8540-fad07b4c628c None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:22:03.803657 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cedf5c7-cde6-47b7-8540-fad07b4c628c None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:22:03.803767 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cedf5c7-cde6-47b7-8540-fad07b4c628c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:22:03.822141 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-9cedf5c7-cde6-47b7-8540-fad07b4c628c None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:22:03.823315 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-9cedf5c7-cde6-47b7-8540-fad07b4c628c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:22:03.857022 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:22:03.889467 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 562/1125] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:22:03 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 29 21:22:04.158970 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff061057-5023-4425-9c41-851e9a20311c None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:22:04.162738 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:22:04.257920 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff061057-5023-4425-9c41-851e9a20311c None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:22:04.290694 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ddfe1af-bfef-4a68-b807-dc9f7fd4029c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:22:04.291311 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ddfe1af-bfef-4a68-b807-dc9f7fd4029c None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:22:04.291587 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ddfe1af-bfef-4a68-b807-dc9f7fd4029c None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:22:04.291856 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ddfe1af-bfef-4a68-b807-dc9f7fd4029c None admin] PATH_INFO: `/v3/roles/9745586777bf473bab7d887bf25bf236` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:22:04.293244 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ddfe1af-bfef-4a68-b807-dc9f7fd4029c None admin] RBAC: Authorizing `identity:get_role(role_id=9745586777bf473bab7d887bf25bf236)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:22:04.294698 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ddfe1af-bfef-4a68-b807-dc9f7fd4029c None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:22:04.295996 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 564/1126] 10.222.0.90 () {56 vars in 1177 bytes} [Mon Aug 29 21:22:04 2022] GET /identity/v3/roles/9745586777bf473bab7d887bf25bf236 => generated 223 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:22:04.302398 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cedf5c7-cde6-47b7-8540-fad07b4c628c None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:22:04.399414 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cedf5c7-cde6-47b7-8540-fad07b4c628c None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:22:04.433508 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49d64e6b-fd07-42fa-9d2f-1d4e573ff498 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:22:04.434102 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49d64e6b-fd07-42fa-9d2f-1d4e573ff498 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:22:04.434384 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49d64e6b-fd07-42fa-9d2f-1d4e573ff498 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:22:04.434656 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49d64e6b-fd07-42fa-9d2f-1d4e573ff498 None admin] PATH_INFO: `/v3/users/86790606a1a848088b14ac17e2a21d71` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:22:04.435769 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49d64e6b-fd07-42fa-9d2f-1d4e573ff498 None admin] RBAC: Authorizing `identity:get_user(user_id=86790606a1a848088b14ac17e2a21d71)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:22:04.437196 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49d64e6b-fd07-42fa-9d2f-1d4e573ff498 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:22:04.438922 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 563/1127] 10.222.0.90 () {56 vars in 1177 bytes} [Mon Aug 29 21:22:04 2022] GET /identity/v3/users/86790606a1a848088b14ac17e2a21d71 => generated 286 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:22:04.444551 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ddfe1af-bfef-4a68-b807-dc9f7fd4029c None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:22:04.513663 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:22:04.543244 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ddfe1af-bfef-4a68-b807-dc9f7fd4029c None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:22:04.575655 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8c22cd8-0c26-4062-9858-4f66d7e4f821 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:22:04.576251 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8c22cd8-0c26-4062-9858-4f66d7e4f821 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:22:04.576528 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8c22cd8-0c26-4062-9858-4f66d7e4f821 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:22:04.576794 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8c22cd8-0c26-4062-9858-4f66d7e4f821 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:22:04.642838 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8c22cd8-0c26-4062-9858-4f66d7e4f821 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:22:04.644267 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8c22cd8-0c26-4062-9858-4f66d7e4f821 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:22:04.709870 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:22:04.740395 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 565/1128] 10.222.0.90 () {58 vars in 1282 bytes} [Mon Aug 29 21:22:04 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 296 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 29 21:22:04.746243 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49d64e6b-fd07-42fa-9d2f-1d4e573ff498 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:22:04.787265 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:22:04.844668 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49d64e6b-fd07-42fa-9d2f-1d4e573ff498 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:22:04.879709 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79dcff12-490b-4614-b200-a974c75cde15 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:22:04.880344 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79dcff12-490b-4614-b200-a974c75cde15 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:22:04.880625 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79dcff12-490b-4614-b200-a974c75cde15 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:22:04.880897 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79dcff12-490b-4614-b200-a974c75cde15 None admin] PATH_INFO: `/v3/projects/4e31c4f75dea41809073912235f7ab55` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:22:04.882040 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79dcff12-490b-4614-b200-a974c75cde15 None admin] RBAC: Authorizing `identity:get_project(project_id=4e31c4f75dea41809073912235f7ab55)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:22:04.883476 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79dcff12-490b-4614-b200-a974c75cde15 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:22:04.885110 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 564/1129] 10.222.0.90 () {56 vars in 1186 bytes} [Mon Aug 29 21:22:04 2022] GET /identity/v3/projects/4e31c4f75dea41809073912235f7ab55 => generated 311 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:22:04.891967 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8c22cd8-0c26-4062-9858-4f66d7e4f821 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:22:04.990430 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8c22cd8-0c26-4062-9858-4f66d7e4f821 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:22:05.024719 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cdc6945-8e42-4919-896f-0f19f069759d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:22:05.025330 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cdc6945-8e42-4919-896f-0f19f069759d None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:22:05.025687 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cdc6945-8e42-4919-896f-0f19f069759d None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:22:05.026512 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cdc6945-8e42-4919-896f-0f19f069759d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:22:05.027222 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cdc6945-8e42-4919-896f-0f19f069759d None admin] RBAC: Adding query filter params (role.id=9745586777bf473bab7d887bf25bf236, scope.project.id=4e31c4f75dea41809073912235f7ab55, user.id=86790606a1a848088b14ac17e2a21d71) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:22:05.027222 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cdc6945-8e42-4919-896f-0f19f069759d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:22:05.028406 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cdc6945-8e42-4919-896f-0f19f069759d None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:22:05.035615 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 566/1130] 10.222.0.90 () {56 vars in 1374 bytes} [Mon Aug 29 21:22:04 2022] GET /identity/v3/role_assignments?user.id=86790606a1a848088b14ac17e2a21d71&scope.project.id=4e31c4f75dea41809073912235f7ab55&role.id=9745586777bf473bab7d887bf25bf236 => generated 608 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:22:06.943013 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ffbd3f3-132a-44dc-b5e4-bee39b7efbdc None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:22:06.943552 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ffbd3f3-132a-44dc-b5e4-bee39b7efbdc None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:22:06.943552 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ffbd3f3-132a-44dc-b5e4-bee39b7efbdc None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:22:06.944890 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 565/1131] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:22:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:22:06.950359 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-491c26f7-ef35-4156-b535-b036ba46b73f None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:22:06.950775 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-491c26f7-ef35-4156-b535-b036ba46b73f None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:22:06.950911 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-491c26f7-ef35-4156-b535-b036ba46b73f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:22:06.969995 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-491c26f7-ef35-4156-b535-b036ba46b73f None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:22:06.971443 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-491c26f7-ef35-4156-b535-b036ba46b73f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:22:06.999383 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:22:07.038219 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 567/1132] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:22:06 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 21:22:07.308995 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ffbd3f3-132a-44dc-b5e4-bee39b7efbdc None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:22:07.407729 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ffbd3f3-132a-44dc-b5e4-bee39b7efbdc None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:22:07.416564 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:22:07.442391 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48eb6dfd-ab9e-434f-8271-c6498c582692 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:22:07.443018 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48eb6dfd-ab9e-434f-8271-c6498c582692 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:22:07.443300 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48eb6dfd-ab9e-434f-8271-c6498c582692 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:22:07.443574 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48eb6dfd-ab9e-434f-8271-c6498c582692 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:22:07.444684 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48eb6dfd-ab9e-434f-8271-c6498c582692 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:22:07.446141 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48eb6dfd-ab9e-434f-8271-c6498c582692 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:22:07.447693 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 566/1133] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:22:07 2022] GET /identity/v3/domains/default => generated 199 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:22:07.453503 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-491c26f7-ef35-4156-b535-b036ba46b73f None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:22:07.551089 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-491c26f7-ef35-4156-b535-b036ba46b73f None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:22:07.584793 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-542a6ca5-bd9d-493e-bf83-f1a80821fef4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:22:07.585372 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-542a6ca5-bd9d-493e-bf83-f1a80821fef4 None admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:22:07.585686 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-542a6ca5-bd9d-493e-bf83-f1a80821fef4 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:22:07.586006 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-542a6ca5-bd9d-493e-bf83-f1a80821fef4 None admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:22:07.586670 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-542a6ca5-bd9d-493e-bf83-f1a80821fef4 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:22:07.588201 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-542a6ca5-bd9d-493e-bf83-f1a80821fef4 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:22:07.615659 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 568/1134] 10.222.0.90 () {60 vars in 1141 bytes} [Mon Aug 29 21:22:07 2022] POST /identity/v3/projects => generated 311 bytes in 163 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 21:22:09.521257 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-000a0fd5-cbe6-464f-9341-19434b578edd None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:22:09.521841 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-000a0fd5-cbe6-464f-9341-19434b578edd None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:22:09.521841 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-000a0fd5-cbe6-464f-9341-19434b578edd None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:22:09.523136 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 567/1135] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:22:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:22:09.528354 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e3daa2d-a5e4-499f-b516-dff8e9e59779 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:22:09.528782 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e3daa2d-a5e4-499f-b516-dff8e9e59779 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:22:09.528899 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e3daa2d-a5e4-499f-b516-dff8e9e59779 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:22:09.547897 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-9e3daa2d-a5e4-499f-b516-dff8e9e59779 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:22:09.549065 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-9e3daa2d-a5e4-499f-b516-dff8e9e59779 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:22:09.561693 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:22:09.617195 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 569/1136] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:22:09 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 29 21:22:09.883015 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-000a0fd5-cbe6-464f-9341-19434b578edd None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:22:09.982414 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-000a0fd5-cbe6-464f-9341-19434b578edd None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:22:10.016163 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffeb52fe-96cc-4122-99c2-40fe2eacf91a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:22:10.016816 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffeb52fe-96cc-4122-99c2-40fe2eacf91a None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:22:10.017085 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffeb52fe-96cc-4122-99c2-40fe2eacf91a None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:22:10.017351 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffeb52fe-96cc-4122-99c2-40fe2eacf91a None admin] PATH_INFO: `/v3/domains/default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:22:10.018475 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffeb52fe-96cc-4122-99c2-40fe2eacf91a None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:22:10.019890 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffeb52fe-96cc-4122-99c2-40fe2eacf91a None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:22:10.021443 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 568/1137] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:22:09 2022] GET /identity/v3/domains/default => generated 199 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:22:10.027334 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e3daa2d-a5e4-499f-b516-dff8e9e59779 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:22:10.125429 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e3daa2d-a5e4-499f-b516-dff8e9e59779 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:22:10.159196 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63e3f666-ba0a-44a7-a012-7cc31760c6fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:22:10.159799 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63e3f666-ba0a-44a7-a012-7cc31760c6fc None admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:22:10.160080 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63e3f666-ba0a-44a7-a012-7cc31760c6fc None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:22:10.160350 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63e3f666-ba0a-44a7-a012-7cc31760c6fc None admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:22:10.161009 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63e3f666-ba0a-44a7-a012-7cc31760c6fc None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:22:10.162467 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63e3f666-ba0a-44a7-a012-7cc31760c6fc None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:22:10.182858 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 570/1138] 10.222.0.90 () {60 vars in 1132 bytes} [Mon Aug 29 21:22:10 2022] POST /identity/v3/users => generated 286 bytes in 156 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:22:12.122985 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c508894e-f3e4-440e-9099-713543d03652 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:22:12.123502 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c508894e-f3e4-440e-9099-713543d03652 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:22:12.123502 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c508894e-f3e4-440e-9099-713543d03652 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:22:12.124817 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 569/1139] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:22:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:22:12.129987 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be5ffb1-56dd-45e1-875e-6ee3ce6aee31 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:22:12.130409 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be5ffb1-56dd-45e1-875e-6ee3ce6aee31 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:22:12.130535 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be5ffb1-56dd-45e1-875e-6ee3ce6aee31 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:22:12.137769 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:22:12.149954 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-8be5ffb1-56dd-45e1-875e-6ee3ce6aee31 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:22:12.151152 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-8be5ffb1-56dd-45e1-875e-6ee3ce6aee31 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:22:12.216287 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 571/1140] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:22:12 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 21:22:12.482220 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c508894e-f3e4-440e-9099-713543d03652 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:22:12.529747 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:22:12.584584 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c508894e-f3e4-440e-9099-713543d03652 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:22:12.617283 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cdf3594-f713-4da9-ad9a-e6a082898d3f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:22:12.617921 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cdf3594-f713-4da9-ad9a-e6a082898d3f None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:22:12.618202 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cdf3594-f713-4da9-ad9a-e6a082898d3f None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:22:12.618473 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cdf3594-f713-4da9-ad9a-e6a082898d3f None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:22:12.628768 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cdf3594-f713-4da9-ad9a-e6a082898d3f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 21:22:12.633526 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-8cdf3594-f713-4da9-ad9a-e6a082898d3f None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 21:22:12.633526 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:22:12.633526 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:22:12.633526 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:22:12.633526 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:22:12.633526 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:22:12.633526 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:22:12.633526 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:22:12.633526 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:22:12.633526 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:22:12.633526 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:22:12.633526 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:22:12.633526 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 21:22:12.633526 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 21:22:12.633526 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 21:22:12.633526 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 21:22:12.634770 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 21:22:12.634770 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 21:22:12.634770 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 21:22:12.634770 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 21:22:12.634770 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:22:12.634770 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:22:12.634770 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:22:12.634770 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:22:12.634770 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:22:12.634770 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:22:12.634770 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:22:12.634770 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:22:12.634770 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:22:12.634770 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:22:12.634770 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:22:12.634770 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:22:12.635916 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:22:12.635916 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:22:12.635916 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:22:12.635916 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:22:12.635916 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 21:22:12.635916 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 21:22:12.635916 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 21:22:12.635916 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 21:22:12.635916 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 21:22:12.635916 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 21:22:12.635916 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 21:22:12.635916 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:22:12.635916 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 570/1141] 10.222.0.90 () {56 vars in 1096 bytes} [Mon Aug 29 21:22:12 2022] GET /identity/v3/roles/admin => generated 83 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:22:12.640807 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8be5ffb1-56dd-45e1-875e-6ee3ce6aee31 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:22:12.737519 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8be5ffb1-56dd-45e1-875e-6ee3ce6aee31 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:22:12.771262 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2599c929-dbd5-4443-a7da-e001535702f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:22:12.771862 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2599c929-dbd5-4443-a7da-e001535702f4 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:22:12.772141 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2599c929-dbd5-4443-a7da-e001535702f4 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:22:12.772409 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2599c929-dbd5-4443-a7da-e001535702f4 None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:22:12.773063 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2599c929-dbd5-4443-a7da-e001535702f4 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:22:12.773361 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2599c929-dbd5-4443-a7da-e001535702f4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:22:12.774745 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2599c929-dbd5-4443-a7da-e001535702f4 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:22:12.776068 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:22:12.783696 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 572/1142] 10.222.0.90 () {56 vars in 1099 bytes} [Mon Aug 29 21:22:12 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:22:12.789625 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cdf3594-f713-4da9-ad9a-e6a082898d3f None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:22:12.889493 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cdf3594-f713-4da9-ad9a-e6a082898d3f None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:22:12.892052 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:22:12.924542 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-485928df-7839-4658-b3f4-b0ca9e3a667d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:22:12.925169 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-485928df-7839-4658-b3f4-b0ca9e3a667d None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:22:12.925452 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-485928df-7839-4658-b3f4-b0ca9e3a667d None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:22:12.925724 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-485928df-7839-4658-b3f4-b0ca9e3a667d None admin] PATH_INFO: `/v3/users/be78c60441034de6a5f5a60916b69914` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:22:12.939310 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-485928df-7839-4658-b3f4-b0ca9e3a667d None admin] RBAC: Authorizing `identity:get_user(user_id=be78c60441034de6a5f5a60916b69914)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:22:12.940729 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-485928df-7839-4658-b3f4-b0ca9e3a667d None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:22:12.942342 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 571/1143] 10.222.0.90 () {56 vars in 1177 bytes} [Mon Aug 29 21:22:12 2022] GET /identity/v3/users/be78c60441034de6a5f5a60916b69914 => generated 286 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:22:12.948226 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2599c929-dbd5-4443-a7da-e001535702f4 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:22:13.047376 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2599c929-dbd5-4443-a7da-e001535702f4 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:22:13.079966 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42569108-864e-4b7c-8afc-70fc73715918 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:22:13.080561 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42569108-864e-4b7c-8afc-70fc73715918 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:22:13.080837 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42569108-864e-4b7c-8afc-70fc73715918 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:22:13.081103 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42569108-864e-4b7c-8afc-70fc73715918 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:22:13.141738 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:22:13.148741 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42569108-864e-4b7c-8afc-70fc73715918 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:22:13.150216 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42569108-864e-4b7c-8afc-70fc73715918 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:22:13.246829 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 573/1144] 10.222.0.90 () {58 vars in 1282 bytes} [Mon Aug 29 21:22:12 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 299 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 29 21:22:13.252855 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-485928df-7839-4658-b3f4-b0ca9e3a667d None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:22:13.350607 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-485928df-7839-4658-b3f4-b0ca9e3a667d None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:22:13.383454 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63673122-1fce-439d-9148-5bd4bec43dbb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:22:13.384073 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63673122-1fce-439d-9148-5bd4bec43dbb None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:22:13.384354 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63673122-1fce-439d-9148-5bd4bec43dbb None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:22:13.384630 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63673122-1fce-439d-9148-5bd4bec43dbb None admin] PATH_INFO: `/v3/projects/be175cbb87e74702a6ba779f6acd47b5` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:22:13.385743 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63673122-1fce-439d-9148-5bd4bec43dbb None admin] RBAC: Authorizing `identity:get_project(project_id=be175cbb87e74702a6ba779f6acd47b5)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:22:13.387194 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63673122-1fce-439d-9148-5bd4bec43dbb None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:22:13.388798 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 572/1145] 10.222.0.90 () {56 vars in 1186 bytes} [Mon Aug 29 21:22:13 2022] GET /identity/v3/projects/be175cbb87e74702a6ba779f6acd47b5 => generated 311 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:22:13.395489 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42569108-864e-4b7c-8afc-70fc73715918 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:22:13.487017 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:22:13.493027 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42569108-864e-4b7c-8afc-70fc73715918 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:22:13.526605 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0767637e-0270-4a63-ba20-93c37d663379 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:22:13.527201 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0767637e-0270-4a63-ba20-93c37d663379 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:22:13.527484 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0767637e-0270-4a63-ba20-93c37d663379 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:22:13.527759 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0767637e-0270-4a63-ba20-93c37d663379 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:22:13.528483 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0767637e-0270-4a63-ba20-93c37d663379 None admin] RBAC: Adding query filter params (role.id=a6e562cdc96c4be691d638990f8403fd, scope.project.id=be175cbb87e74702a6ba779f6acd47b5, user.id=be78c60441034de6a5f5a60916b69914) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:22:13.528791 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0767637e-0270-4a63-ba20-93c37d663379 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:22:13.530236 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0767637e-0270-4a63-ba20-93c37d663379 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:22:13.535552 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 574/1146] 10.222.0.90 () {56 vars in 1374 bytes} [Mon Aug 29 21:22:13 2022] GET /identity/v3/role_assignments?user.id=be78c60441034de6a5f5a60916b69914&scope.project.id=be175cbb87e74702a6ba779f6acd47b5&role.id=a6e562cdc96c4be691d638990f8403fd => generated 259 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:22:15.416262 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76805758-005c-415f-8e23-bb1370c5627a None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:22:15.416800 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76805758-005c-415f-8e23-bb1370c5627a None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:22:15.416918 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76805758-005c-415f-8e23-bb1370c5627a None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:22:15.418282 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 573/1147] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:22:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:22:15.424422 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e5835f8-ed1b-47be-a25d-197a1935e7e4 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:22:15.424831 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e5835f8-ed1b-47be-a25d-197a1935e7e4 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:22:15.424973 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e5835f8-ed1b-47be-a25d-197a1935e7e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:22:15.442759 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-2e5835f8-ed1b-47be-a25d-197a1935e7e4 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:22:15.443933 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-2e5835f8-ed1b-47be-a25d-197a1935e7e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:22:15.511272 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 575/1148] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:22:15 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 21:22:15.779557 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76805758-005c-415f-8e23-bb1370c5627a None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:22:15.811004 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:22:15.883489 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76805758-005c-415f-8e23-bb1370c5627a None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:22:15.917628 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a9bb59a-1411-4df7-86d9-09992116ec85 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:22:15.918277 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a9bb59a-1411-4df7-86d9-09992116ec85 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:22:15.918552 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a9bb59a-1411-4df7-86d9-09992116ec85 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:22:15.918821 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a9bb59a-1411-4df7-86d9-09992116ec85 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:22:15.928920 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a9bb59a-1411-4df7-86d9-09992116ec85 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 21:22:15.933677 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-0a9bb59a-1411-4df7-86d9-09992116ec85 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 21:22:15.933677 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:22:15.933677 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:22:15.933677 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:22:15.933677 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:22:15.933677 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:22:15.933677 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:22:15.933677 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:22:15.933677 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:22:15.933677 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:22:15.933677 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:22:15.933677 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:22:15.933677 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 21:22:15.933677 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 21:22:15.933677 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 21:22:15.933677 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 21:22:15.934846 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 21:22:15.934846 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 21:22:15.934846 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 21:22:15.934846 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 21:22:15.934846 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:22:15.934846 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:22:15.934846 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:22:15.934846 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:22:15.934846 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:22:15.934846 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:22:15.934846 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:22:15.934846 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:22:15.934846 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:22:15.934846 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:22:15.934846 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:22:15.934846 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:22:15.935731 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:22:15.935731 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:22:15.935731 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:22:15.935731 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:22:15.935731 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 21:22:15.935731 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 21:22:15.935731 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 21:22:15.935731 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 21:22:15.935731 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 21:22:15.935731 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 21:22:15.935731 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 21:22:15.935731 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:22:15.935731 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 574/1149] 10.222.0.90 () {56 vars in 1096 bytes} [Mon Aug 29 21:22:15 2022] GET /identity/v3/roles/admin => generated 83 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:22:15.940843 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e5835f8-ed1b-47be-a25d-197a1935e7e4 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:22:16.028365 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:22:16.038641 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e5835f8-ed1b-47be-a25d-197a1935e7e4 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:22:16.071886 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c26fe046-3031-40ec-a3ac-b7498b1ddec3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:22:16.072486 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c26fe046-3031-40ec-a3ac-b7498b1ddec3 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:22:16.072764 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c26fe046-3031-40ec-a3ac-b7498b1ddec3 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:22:16.073128 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c26fe046-3031-40ec-a3ac-b7498b1ddec3 None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:22:16.073792 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c26fe046-3031-40ec-a3ac-b7498b1ddec3 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:22:16.074109 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c26fe046-3031-40ec-a3ac-b7498b1ddec3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:22:16.075469 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c26fe046-3031-40ec-a3ac-b7498b1ddec3 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:22:16.083369 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 576/1150] 10.222.0.90 () {56 vars in 1099 bytes} [Mon Aug 29 21:22:15 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:22:16.090414 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a9bb59a-1411-4df7-86d9-09992116ec85 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:22:16.180048 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:22:16.189675 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a9bb59a-1411-4df7-86d9-09992116ec85 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:22:16.224134 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d98df3c6-c620-417d-9306-670bd0699b6a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:22:16.224765 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d98df3c6-c620-417d-9306-670bd0699b6a None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:22:16.225048 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d98df3c6-c620-417d-9306-670bd0699b6a None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:22:16.225325 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d98df3c6-c620-417d-9306-670bd0699b6a None admin] PATH_INFO: `/v3/users/be78c60441034de6a5f5a60916b69914` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:22:16.226518 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d98df3c6-c620-417d-9306-670bd0699b6a None admin] RBAC: Authorizing `identity:get_user(user_id=be78c60441034de6a5f5a60916b69914)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:22:16.227950 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d98df3c6-c620-417d-9306-670bd0699b6a None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:22:16.229515 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 575/1151] 10.222.0.90 () {56 vars in 1177 bytes} [Mon Aug 29 21:22:16 2022] GET /identity/v3/users/be78c60441034de6a5f5a60916b69914 => generated 286 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:22:16.236084 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c26fe046-3031-40ec-a3ac-b7498b1ddec3 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:22:16.333629 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c26fe046-3031-40ec-a3ac-b7498b1ddec3 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:22:16.366696 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8bc0239-8c8f-4838-869c-bac9f6158bd2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:22:16.367306 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8bc0239-8c8f-4838-869c-bac9f6158bd2 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:22:16.367581 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8bc0239-8c8f-4838-869c-bac9f6158bd2 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:22:16.367850 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8bc0239-8c8f-4838-869c-bac9f6158bd2 None admin] PATH_INFO: `/v3/projects/be175cbb87e74702a6ba779f6acd47b5` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:22:16.368950 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8bc0239-8c8f-4838-869c-bac9f6158bd2 None admin] RBAC: Authorizing `identity:get_project(project_id=be175cbb87e74702a6ba779f6acd47b5)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:22:16.370406 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8bc0239-8c8f-4838-869c-bac9f6158bd2 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:22:16.372009 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 577/1152] 10.222.0.90 () {56 vars in 1186 bytes} [Mon Aug 29 21:22:16 2022] GET /identity/v3/projects/be175cbb87e74702a6ba779f6acd47b5 => generated 311 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:22:16.378029 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d98df3c6-c620-417d-9306-670bd0699b6a None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:22:16.474665 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d98df3c6-c620-417d-9306-670bd0699b6a None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:22:16.507191 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88e658cb-ac8f-44eb-96db-a9fe36192f21 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:22:16.507835 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88e658cb-ac8f-44eb-96db-a9fe36192f21 None admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:22:16.508146 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88e658cb-ac8f-44eb-96db-a9fe36192f21 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:22:16.508422 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88e658cb-ac8f-44eb-96db-a9fe36192f21 None admin] PATH_INFO: `/v3/projects/be175cbb87e74702a6ba779f6acd47b5/users/be78c60441034de6a5f5a60916b69914/roles/a6e562cdc96c4be691d638990f8403fd` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:22:16.510339 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88e658cb-ac8f-44eb-96db-a9fe36192f21 None admin] RBAC: Authorizing `identity:create_grant(project_id=be175cbb87e74702a6ba779f6acd47b5, user_id=be78c60441034de6a5f5a60916b69914, role_id=a6e562cdc96c4be691d638990f8403fd)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:22:16.511758 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88e658cb-ac8f-44eb-96db-a9fe36192f21 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:22:16.524854 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 576/1153] 10.222.0.90 () {58 vars in 1439 bytes} [Mon Aug 29 21:22:16 2022] PUT /identity/v3/projects/be175cbb87e74702a6ba779f6acd47b5/users/be78c60441034de6a5f5a60916b69914/roles/a6e562cdc96c4be691d638990f8403fd => generated 0 bytes in 147 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:22:18.416928 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d99753cc-152c-4d92-a0bf-d20825262dfe None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:22:18.417466 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d99753cc-152c-4d92-a0bf-d20825262dfe None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:22:18.417466 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d99753cc-152c-4d92-a0bf-d20825262dfe None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:22:18.418845 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 578/1154] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:22:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:22:18.424523 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af1b5da6-79f1-40ae-8aa6-87b1da32457c None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:22:18.424958 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af1b5da6-79f1-40ae-8aa6-87b1da32457c None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:22:18.425053 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af1b5da6-79f1-40ae-8aa6-87b1da32457c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:22:18.443029 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-af1b5da6-79f1-40ae-8aa6-87b1da32457c None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:22:18.444221 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-af1b5da6-79f1-40ae-8aa6-87b1da32457c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:22:18.457224 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:22:18.511647 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 577/1155] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:22:18 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 29 21:22:18.779685 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d99753cc-152c-4d92-a0bf-d20825262dfe None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:22:18.802989 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:22:18.883762 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d99753cc-152c-4d92-a0bf-d20825262dfe None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:22:18.916086 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26822708-920a-4a8f-86ec-c88df000f8f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:22:18.916714 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26822708-920a-4a8f-86ec-c88df000f8f2 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:22:18.916999 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26822708-920a-4a8f-86ec-c88df000f8f2 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:22:18.917277 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26822708-920a-4a8f-86ec-c88df000f8f2 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:22:18.928300 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26822708-920a-4a8f-86ec-c88df000f8f2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 21:22:18.933077 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-26822708-920a-4a8f-86ec-c88df000f8f2 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 21:22:18.933077 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:22:18.933077 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:22:18.933077 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:22:18.933077 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:22:18.933077 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:22:18.933077 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:22:18.933077 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:22:18.933077 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:22:18.933077 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:22:18.933077 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:22:18.933077 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:22:18.933077 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 21:22:18.933077 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 21:22:18.933077 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 21:22:18.933077 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 21:22:18.934251 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 21:22:18.934251 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 21:22:18.934251 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 21:22:18.934251 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 21:22:18.934251 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:22:18.934251 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:22:18.934251 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:22:18.934251 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:22:18.934251 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:22:18.934251 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:22:18.934251 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:22:18.934251 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:22:18.934251 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:22:18.934251 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:22:18.934251 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:22:18.934251 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:22:18.935591 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:22:18.935591 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:22:18.935591 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:22:18.935591 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:22:18.935591 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 21:22:18.935591 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 21:22:18.935591 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 21:22:18.935591 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 21:22:18.935591 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 21:22:18.935591 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 21:22:18.935591 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 21:22:18.935591 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:22:18.935591 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 579/1156] 10.222.0.90 () {56 vars in 1096 bytes} [Mon Aug 29 21:22:18 2022] GET /identity/v3/roles/admin => generated 83 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:22:18.941284 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af1b5da6-79f1-40ae-8aa6-87b1da32457c None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:22:19.039769 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af1b5da6-79f1-40ae-8aa6-87b1da32457c None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:22:19.073651 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-386b7d06-f350-4933-8cda-7b4fcbb1eaec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:22:19.074286 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-386b7d06-f350-4933-8cda-7b4fcbb1eaec None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:22:19.074568 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-386b7d06-f350-4933-8cda-7b4fcbb1eaec None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:22:19.074843 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-386b7d06-f350-4933-8cda-7b4fcbb1eaec None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:22:19.075504 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-386b7d06-f350-4933-8cda-7b4fcbb1eaec None admin] RBAC: Adding query filter params (name=admin) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:22:19.075835 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-386b7d06-f350-4933-8cda-7b4fcbb1eaec None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:22:19.077201 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-386b7d06-f350-4933-8cda-7b4fcbb1eaec None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:22:19.085235 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 578/1157] 10.222.0.90 () {56 vars in 1099 bytes} [Mon Aug 29 21:22:18 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:22:19.091034 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26822708-920a-4a8f-86ec-c88df000f8f2 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:22:19.170635 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:22:19.190948 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26822708-920a-4a8f-86ec-c88df000f8f2 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:22:19.224125 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47498e93-e573-4788-95ee-195d743649d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:22:19.224739 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47498e93-e573-4788-95ee-195d743649d0 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:22:19.225017 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47498e93-e573-4788-95ee-195d743649d0 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:22:19.225288 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47498e93-e573-4788-95ee-195d743649d0 None admin] PATH_INFO: `/v3/users/be78c60441034de6a5f5a60916b69914` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:22:19.226403 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47498e93-e573-4788-95ee-195d743649d0 None admin] RBAC: Authorizing `identity:get_user(user_id=be78c60441034de6a5f5a60916b69914)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:22:19.227819 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47498e93-e573-4788-95ee-195d743649d0 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:22:19.229471 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 580/1158] 10.222.0.90 () {56 vars in 1177 bytes} [Mon Aug 29 21:22:19 2022] GET /identity/v3/users/be78c60441034de6a5f5a60916b69914 => generated 286 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:22:19.236169 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-386b7d06-f350-4933-8cda-7b4fcbb1eaec None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:22:19.248717 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:22:19.334084 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-386b7d06-f350-4933-8cda-7b4fcbb1eaec None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:22:19.367923 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc6a9b3f-e100-4837-a491-d4629fb4fd21 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:22:19.368524 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc6a9b3f-e100-4837-a491-d4629fb4fd21 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:22:19.368806 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc6a9b3f-e100-4837-a491-d4629fb4fd21 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:22:19.369083 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc6a9b3f-e100-4837-a491-d4629fb4fd21 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:22:19.434333 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc6a9b3f-e100-4837-a491-d4629fb4fd21 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:22:19.435765 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc6a9b3f-e100-4837-a491-d4629fb4fd21 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:22:19.446883 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:22:19.535335 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 579/1159] 10.222.0.90 () {58 vars in 1282 bytes} [Mon Aug 29 21:22:19 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 300 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 29 21:22:19.541249 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47498e93-e573-4788-95ee-195d743649d0 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:22:19.637469 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47498e93-e573-4788-95ee-195d743649d0 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:22:19.671108 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3a57df7-5988-4df0-b170-525edc010cb9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:22:19.671791 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3a57df7-5988-4df0-b170-525edc010cb9 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:22:19.672072 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3a57df7-5988-4df0-b170-525edc010cb9 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:22:19.672348 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3a57df7-5988-4df0-b170-525edc010cb9 None admin] PATH_INFO: `/v3/projects/be175cbb87e74702a6ba779f6acd47b5` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:22:19.673468 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3a57df7-5988-4df0-b170-525edc010cb9 None admin] RBAC: Authorizing `identity:get_project(project_id=be175cbb87e74702a6ba779f6acd47b5)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:22:19.674932 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3a57df7-5988-4df0-b170-525edc010cb9 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:22:19.676557 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 581/1160] 10.222.0.90 () {56 vars in 1186 bytes} [Mon Aug 29 21:22:19 2022] GET /identity/v3/projects/be175cbb87e74702a6ba779f6acd47b5 => generated 311 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:22:19.682626 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc6a9b3f-e100-4837-a491-d4629fb4fd21 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:22:19.779938 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc6a9b3f-e100-4837-a491-d4629fb4fd21 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:22:19.790688 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:22:19.814066 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02fa720d-0bc5-4da6-878b-fe18429bd218 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:22:19.814671 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02fa720d-0bc5-4da6-878b-fe18429bd218 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:22:19.814950 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02fa720d-0bc5-4da6-878b-fe18429bd218 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:22:19.815221 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02fa720d-0bc5-4da6-878b-fe18429bd218 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:22:19.815944 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02fa720d-0bc5-4da6-878b-fe18429bd218 None admin] RBAC: Adding query filter params (role.id=a6e562cdc96c4be691d638990f8403fd, scope.project.id=be175cbb87e74702a6ba779f6acd47b5, user.id=be78c60441034de6a5f5a60916b69914) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:22:19.816248 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02fa720d-0bc5-4da6-878b-fe18429bd218 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:22:19.817670 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02fa720d-0bc5-4da6-878b-fe18429bd218 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:22:19.823263 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 580/1161] 10.222.0.90 () {56 vars in 1374 bytes} [Mon Aug 29 21:22:19 2022] GET /identity/v3/role_assignments?user.id=be78c60441034de6a5f5a60916b69914&scope.project.id=be175cbb87e74702a6ba779f6acd47b5&role.id=a6e562cdc96c4be691d638990f8403fd => generated 608 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:22:21.749450 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3613cba1-59f3-4f81-926e-9c16ea638907 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:22:21.750065 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3613cba1-59f3-4f81-926e-9c16ea638907 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:22:21.750065 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3613cba1-59f3-4f81-926e-9c16ea638907 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:22:21.751366 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 582/1162] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:22:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:22:21.756541 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff1cd265-11c9-4873-8c60-129064f86289 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:22:21.756914 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff1cd265-11c9-4873-8c60-129064f86289 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:22:21.757097 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff1cd265-11c9-4873-8c60-129064f86289 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:22:21.775520 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-ff1cd265-11c9-4873-8c60-129064f86289 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:22:21.776712 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-ff1cd265-11c9-4873-8c60-129064f86289 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:22:21.844249 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 581/1163] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:22:21 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 21:22:22.123704 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3613cba1-59f3-4f81-926e-9c16ea638907 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:22:22.147487 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:22:22.231355 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3613cba1-59f3-4f81-926e-9c16ea638907 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:22:22.265510 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90ab2da8-de66-4f30-a920-ca930d22fc24 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:22:22.266145 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90ab2da8-de66-4f30-a920-ca930d22fc24 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:22:22.266429 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90ab2da8-de66-4f30-a920-ca930d22fc24 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:22:22.266701 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90ab2da8-de66-4f30-a920-ca930d22fc24 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:22:22.333195 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90ab2da8-de66-4f30-a920-ca930d22fc24 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:22:22.334662 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90ab2da8-de66-4f30-a920-ca930d22fc24 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:22:22.351229 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:22:22.431354 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 583/1164] 10.222.0.90 () {58 vars in 1282 bytes} [Mon Aug 29 21:22:22 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Aug 29 21:22:22.438091 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff1cd265-11c9-4873-8c60-129064f86289 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:22:22.534114 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff1cd265-11c9-4873-8c60-129064f86289 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:22:22.536415 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:22:22.568143 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d87409e4-848a-414c-83f2-3123cf3e1bc7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:22:22.568772 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d87409e4-848a-414c-83f2-3123cf3e1bc7 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:22:22.569054 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d87409e4-848a-414c-83f2-3123cf3e1bc7 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:22:22.569326 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d87409e4-848a-414c-83f2-3123cf3e1bc7 None admin] PATH_INFO: `/v3/domains/swift_test` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:22:22.575795 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d87409e4-848a-414c-83f2-3123cf3e1bc7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=swift_test)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:22:22.577265 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d87409e4-848a-414c-83f2-3123cf3e1bc7 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:22:22.588468 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-d87409e4-848a-414c-83f2-3123cf3e1bc7 None admin] Could not find domain: swift_test.: keystone.exception.DomainNotFound: Could not find domain: swift_test. Aug 29 21:22:22.588468 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:22:22.588468 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 712, in get_domain Aug 29 21:22:22.588468 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application project = self.driver.get_project(domain_id) Aug 29 21:22:22.588468 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 29 21:22:22.588468 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 29 21:22:22.588468 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 29 21:22:22.588468 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 29 21:22:22.588468 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: swift_test. Aug 29 21:22:22.588468 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:22:22.588468 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 29 21:22:22.588468 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:22:22.588468 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:22:22.588468 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:22:22.588468 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:22:22.588468 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:22:22.588468 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:22:22.589712 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:22:22.589712 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:22:22.589712 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:22:22.589712 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:22:22.589712 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:22:22.589712 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:22:22.589712 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 21:22:22.589712 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 21:22:22.589712 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 21:22:22.589712 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 21:22:22.589712 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:22:22.589712 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:22:22.589712 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:22:22.589712 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:22:22.589712 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:22:22.589712 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:22:22.590641 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:22:22.590641 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:22:22.590641 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:22:22.590641 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:22:22.590641 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:22:22.590641 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:22:22.590641 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:22:22.590641 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:22:22.590641 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:22:22.590641 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:22:22.590641 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 718, in get_domain Aug 29 21:22:22.590641 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 21:22:22.590641 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: swift_test. Aug 29 21:22:22.590641 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:22:22.590641 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 582/1165] 10.222.0.90 () {56 vars in 1117 bytes} [Mon Aug 29 21:22:22 2022] GET /identity/v3/domains/swift_test => generated 90 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:22:22.595953 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90ab2da8-de66-4f30-a920-ca930d22fc24 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:22:22.693514 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90ab2da8-de66-4f30-a920-ca930d22fc24 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:22:22.709143 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:22:22.727176 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1bf9b7b-f178-4387-8ef9-61c01b213eed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:22:22.727777 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1bf9b7b-f178-4387-8ef9-61c01b213eed None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:22:22.728057 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1bf9b7b-f178-4387-8ef9-61c01b213eed None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:22:22.728330 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1bf9b7b-f178-4387-8ef9-61c01b213eed None admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:22:22.728969 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1bf9b7b-f178-4387-8ef9-61c01b213eed None admin] RBAC: Adding query filter params (name=swift_test) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:22:22.729273 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1bf9b7b-f178-4387-8ef9-61c01b213eed None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:22:22.730663 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1bf9b7b-f178-4387-8ef9-61c01b213eed None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:22:22.738053 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 584/1166] 10.222.0.90 () {56 vars in 1115 bytes} [Mon Aug 29 21:22:22 2022] GET /identity/v3/domains?name=swift_test => generated 125 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:22:24.623437 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e03186a3-0f39-45c0-9469-f4d886a47b20 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:22:24.624504 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e03186a3-0f39-45c0-9469-f4d886a47b20 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:22:24.624609 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e03186a3-0f39-45c0-9469-f4d886a47b20 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:22:24.625962 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 583/1167] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:22:24 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 29 21:22:24.631046 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dc1f5d0-ed80-48f0-bf65-ab75ae1a2b7d None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:22:24.631468 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dc1f5d0-ed80-48f0-bf65-ab75ae1a2b7d None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:22:24.631579 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dc1f5d0-ed80-48f0-bf65-ab75ae1a2b7d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:22:24.650395 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-8dc1f5d0-ed80-48f0-bf65-ab75ae1a2b7d None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:22:24.651577 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-8dc1f5d0-ed80-48f0-bf65-ab75ae1a2b7d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:22:24.717501 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 585/1168] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:22:24 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 21:22:24.990484 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e03186a3-0f39-45c0-9469-f4d886a47b20 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:22:25.089506 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e03186a3-0f39-45c0-9469-f4d886a47b20 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:22:25.123254 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15e46210-91a5-4ecc-9d09-d8b6472c7903 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:22:25.123850 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e46210-91a5-4ecc-9d09-d8b6472c7903 None admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:22:25.124501 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e46210-91a5-4ecc-9d09-d8b6472c7903 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:22:25.124747 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e46210-91a5-4ecc-9d09-d8b6472c7903 None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:22:25.125414 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15e46210-91a5-4ecc-9d09-d8b6472c7903 None admin] RBAC: Authorizing `identity:create_domain()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:22:25.126802 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15e46210-91a5-4ecc-9d09-d8b6472c7903 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:22:25.142398 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 584/1169] 10.222.0.90 () {60 vars in 1138 bytes} [Mon Aug 29 21:22:24 2022] POST /identity/v3/domains => generated 267 bytes in 153 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:22:27.080239 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3561dbd-468b-4b1d-ae62-2c112e0d6348 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:22:27.080823 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3561dbd-468b-4b1d-ae62-2c112e0d6348 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:22:27.080823 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3561dbd-468b-4b1d-ae62-2c112e0d6348 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:22:27.082155 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 586/1170] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:22:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:22:27.087458 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeca7ece-0050-4d6c-8a94-7517ef7eb7f0 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:22:27.087906 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeca7ece-0050-4d6c-8a94-7517ef7eb7f0 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:22:27.088012 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeca7ece-0050-4d6c-8a94-7517ef7eb7f0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:22:27.106506 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-eeca7ece-0050-4d6c-8a94-7517ef7eb7f0 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:22:27.107708 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-eeca7ece-0050-4d6c-8a94-7517ef7eb7f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:22:27.125911 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:22:27.174659 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 585/1171] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:22:27 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 29 21:22:27.445365 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3561dbd-468b-4b1d-ae62-2c112e0d6348 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:22:27.544825 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3561dbd-468b-4b1d-ae62-2c112e0d6348 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:22:27.555856 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:22:27.579416 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae6b5590-6df1-43a5-804f-68f68acd85c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:22:27.580043 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae6b5590-6df1-43a5-804f-68f68acd85c9 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:22:27.580325 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae6b5590-6df1-43a5-804f-68f68acd85c9 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:22:27.580599 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae6b5590-6df1-43a5-804f-68f68acd85c9 None admin] PATH_INFO: `/v3/domains/e14ce9b1651c42ecae14ce1d265a8f92` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:22:27.590714 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae6b5590-6df1-43a5-804f-68f68acd85c9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=e14ce9b1651c42ecae14ce1d265a8f92)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:22:27.592158 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae6b5590-6df1-43a5-804f-68f68acd85c9 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:22:27.594327 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 587/1172] 10.222.0.90 () {56 vars in 1183 bytes} [Mon Aug 29 21:22:27 2022] GET /identity/v3/domains/e14ce9b1651c42ecae14ce1d265a8f92 => generated 267 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:22:27.600405 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeca7ece-0050-4d6c-8a94-7517ef7eb7f0 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:22:27.697215 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeca7ece-0050-4d6c-8a94-7517ef7eb7f0 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:22:27.732121 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4030651a-32d3-4fcf-b743-a85b8dfe5d3d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:22:27.732718 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4030651a-32d3-4fcf-b743-a85b8dfe5d3d None admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:22:27.733000 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4030651a-32d3-4fcf-b743-a85b8dfe5d3d None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:22:27.733324 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4030651a-32d3-4fcf-b743-a85b8dfe5d3d None admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:22:27.734291 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4030651a-32d3-4fcf-b743-a85b8dfe5d3d None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:22:27.735719 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4030651a-32d3-4fcf-b743-a85b8dfe5d3d None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:22:27.760919 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 586/1173] 10.222.0.90 () {60 vars in 1141 bytes} [Mon Aug 29 21:22:27 2022] POST /identity/v3/projects => generated 361 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:22:29.677280 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7757d3bb-e752-4e46-89be-b085222bd2d6 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:22:29.677870 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7757d3bb-e752-4e46-89be-b085222bd2d6 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:22:29.677870 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7757d3bb-e752-4e46-89be-b085222bd2d6 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:22:29.679274 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 588/1174] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:22:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:22:29.684547 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c0d35c9-a774-43bb-9f8b-da1ca8e1e71c None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:22:29.684966 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c0d35c9-a774-43bb-9f8b-da1ca8e1e71c None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:22:29.685102 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c0d35c9-a774-43bb-9f8b-da1ca8e1e71c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:22:29.698454 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:22:29.704461 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-1c0d35c9-a774-43bb-9f8b-da1ca8e1e71c None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:22:29.705650 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-1c0d35c9-a774-43bb-9f8b-da1ca8e1e71c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:22:29.772222 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 587/1175] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:22:29 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 21:22:30.050263 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7757d3bb-e752-4e46-89be-b085222bd2d6 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:22:30.152220 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7757d3bb-e752-4e46-89be-b085222bd2d6 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:22:30.185717 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2a1f108-b3d0-46e3-a89a-d4872570ecce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:22:30.186602 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2a1f108-b3d0-46e3-a89a-d4872570ecce None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:22:30.186941 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2a1f108-b3d0-46e3-a89a-d4872570ecce None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:22:30.187235 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2a1f108-b3d0-46e3-a89a-d4872570ecce None admin] PATH_INFO: `/v3/domains/e14ce9b1651c42ecae14ce1d265a8f92` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:22:30.188357 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2a1f108-b3d0-46e3-a89a-d4872570ecce None admin] RBAC: Authorizing `identity:get_domain(domain_id=e14ce9b1651c42ecae14ce1d265a8f92)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:22:30.189805 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2a1f108-b3d0-46e3-a89a-d4872570ecce None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:22:30.191380 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 589/1176] 10.222.0.90 () {56 vars in 1183 bytes} [Mon Aug 29 21:22:30 2022] GET /identity/v3/domains/e14ce9b1651c42ecae14ce1d265a8f92 => generated 267 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:22:30.197074 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c0d35c9-a774-43bb-9f8b-da1ca8e1e71c None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:22:30.295855 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c0d35c9-a774-43bb-9f8b-da1ca8e1e71c None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:22:30.329636 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4660e824-412e-46b4-bf34-94313c646632 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:22:30.330200 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4660e824-412e-46b4-bf34-94313c646632 None admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:22:30.330483 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4660e824-412e-46b4-bf34-94313c646632 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:22:30.330756 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4660e824-412e-46b4-bf34-94313c646632 None admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:22:30.331418 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4660e824-412e-46b4-bf34-94313c646632 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:22:30.332834 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4660e824-412e-46b4-bf34-94313c646632 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:22:30.341626 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:22:30.354473 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 588/1177] 10.222.0.90 () {60 vars in 1132 bytes} [Mon Aug 29 21:22:30 2022] POST /identity/v3/users => generated 311 bytes in 158 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:22:32.309977 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7acb87af-0508-4df6-8197-58088e5fb527 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:22:32.310522 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7acb87af-0508-4df6-8197-58088e5fb527 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:22:32.310522 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7acb87af-0508-4df6-8197-58088e5fb527 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:22:32.312508 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 590/1178] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:22:32 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:22:32.317733 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dd84538-f4be-4485-a79e-8a0216c8ec93 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:22:32.318162 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dd84538-f4be-4485-a79e-8a0216c8ec93 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:22:32.318324 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dd84538-f4be-4485-a79e-8a0216c8ec93 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:22:32.336428 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-7dd84538-f4be-4485-a79e-8a0216c8ec93 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:22:32.337607 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-7dd84538-f4be-4485-a79e-8a0216c8ec93 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:22:32.402001 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 589/1179] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:22:32 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 21:22:32.673064 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7acb87af-0508-4df6-8197-58088e5fb527 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:22:32.714090 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:22:32.774087 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7acb87af-0508-4df6-8197-58088e5fb527 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:22:32.806401 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de5d24a4-b3ba-4746-a166-b15989f595a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:22:32.807021 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de5d24a4-b3ba-4746-a166-b15989f595a5 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:22:32.807300 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de5d24a4-b3ba-4746-a166-b15989f595a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:22:32.807575 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de5d24a4-b3ba-4746-a166-b15989f595a5 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:22:32.818505 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de5d24a4-b3ba-4746-a166-b15989f595a5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 21:22:32.823309 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-de5d24a4-b3ba-4746-a166-b15989f595a5 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 21:22:32.823309 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:22:32.823309 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:22:32.823309 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:22:32.823309 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:22:32.823309 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:22:32.823309 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:22:32.823309 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:22:32.823309 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:22:32.823309 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:22:32.823309 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:22:32.823309 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:22:32.823309 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 21:22:32.823309 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 21:22:32.823309 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 21:22:32.823309 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 21:22:32.824490 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 21:22:32.824490 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 21:22:32.824490 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 21:22:32.824490 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 21:22:32.824490 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:22:32.824490 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:22:32.824490 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:22:32.824490 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:22:32.824490 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:22:32.824490 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:22:32.824490 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:22:32.824490 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:22:32.824490 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:22:32.824490 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:22:32.824490 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:22:32.824490 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:22:32.825474 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:22:32.825474 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:22:32.825474 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:22:32.825474 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:22:32.825474 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 21:22:32.825474 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 21:22:32.825474 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 21:22:32.825474 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 21:22:32.825474 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 21:22:32.825474 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 21:22:32.825474 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 21:22:32.825474 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:22:32.825474 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 591/1180] 10.222.0.90 () {56 vars in 1096 bytes} [Mon Aug 29 21:22:32 2022] GET /identity/v3/roles/admin => generated 83 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:22:32.831040 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dd84538-f4be-4485-a79e-8a0216c8ec93 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:22:32.929243 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dd84538-f4be-4485-a79e-8a0216c8ec93 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:22:32.952428 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:INSERT=2,keystone:SELECT=98 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:22:32.965234 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-668fd31c-e2a3-4088-89ac-35992366093a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:22:32.965906 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-668fd31c-e2a3-4088-89ac-35992366093a None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:22:32.966261 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-668fd31c-e2a3-4088-89ac-35992366093a None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:22:32.966537 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-668fd31c-e2a3-4088-89ac-35992366093a None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:22:32.967225 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-668fd31c-e2a3-4088-89ac-35992366093a None admin] RBAC: Adding query filter params (name=admin) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:22:32.967599 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-668fd31c-e2a3-4088-89ac-35992366093a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:22:32.969745 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-668fd31c-e2a3-4088-89ac-35992366093a None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:22:32.978482 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 590/1181] 10.222.0.90 () {56 vars in 1099 bytes} [Mon Aug 29 21:22:32 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:22:32.985958 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de5d24a4-b3ba-4746-a166-b15989f595a5 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:22:33.082110 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:22:33.086155 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de5d24a4-b3ba-4746-a166-b15989f595a5 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:22:33.119070 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bceb9706-7637-41d2-8fc2-300c781a7067 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:22:33.119678 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bceb9706-7637-41d2-8fc2-300c781a7067 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:22:33.119954 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bceb9706-7637-41d2-8fc2-300c781a7067 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:22:33.120220 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bceb9706-7637-41d2-8fc2-300c781a7067 None admin] PATH_INFO: `/v3/users/0c3f69846e4e4037b7e7165a147c2bde` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:22:33.133429 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bceb9706-7637-41d2-8fc2-300c781a7067 None admin] RBAC: Authorizing `identity:get_user(user_id=0c3f69846e4e4037b7e7165a147c2bde)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:22:33.134878 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bceb9706-7637-41d2-8fc2-300c781a7067 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:22:33.136462 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 592/1182] 10.222.0.90 () {56 vars in 1177 bytes} [Mon Aug 29 21:22:32 2022] GET /identity/v3/users/0c3f69846e4e4037b7e7165a147c2bde => generated 311 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:22:33.142463 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-668fd31c-e2a3-4088-89ac-35992366093a None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:22:33.240063 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-668fd31c-e2a3-4088-89ac-35992366093a None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:22:33.273368 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98a8d200-6fec-4752-9431-6501160cac78 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:22:33.273993 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98a8d200-6fec-4752-9431-6501160cac78 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:22:33.274273 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98a8d200-6fec-4752-9431-6501160cac78 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:22:33.274545 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98a8d200-6fec-4752-9431-6501160cac78 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:22:33.324539 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:22:33.339804 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98a8d200-6fec-4752-9431-6501160cac78 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:22:33.341233 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98a8d200-6fec-4752-9431-6501160cac78 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:22:33.437350 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 591/1183] 10.222.0.90 () {58 vars in 1282 bytes} [Mon Aug 29 21:22:33 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 295 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 29 21:22:33.443364 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bceb9706-7637-41d2-8fc2-300c781a7067 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:22:33.540508 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bceb9706-7637-41d2-8fc2-300c781a7067 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:22:33.573362 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-168cb62b-b38c-46cd-ab7a-eb5f99d3d37e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:22:33.574001 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-168cb62b-b38c-46cd-ab7a-eb5f99d3d37e None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:22:33.574280 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-168cb62b-b38c-46cd-ab7a-eb5f99d3d37e None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:22:33.574548 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-168cb62b-b38c-46cd-ab7a-eb5f99d3d37e None admin] PATH_INFO: `/v3/projects/428e1a82439c463f9a1a57d054223644` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:22:33.575642 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-168cb62b-b38c-46cd-ab7a-eb5f99d3d37e None admin] RBAC: Authorizing `identity:get_project(project_id=428e1a82439c463f9a1a57d054223644)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:22:33.577261 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-168cb62b-b38c-46cd-ab7a-eb5f99d3d37e None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:22:33.578903 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 593/1184] 10.222.0.90 () {56 vars in 1186 bytes} [Mon Aug 29 21:22:33 2022] GET /identity/v3/projects/428e1a82439c463f9a1a57d054223644 => generated 361 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:22:33.584937 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98a8d200-6fec-4752-9431-6501160cac78 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:22:33.668448 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:22:33.683949 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98a8d200-6fec-4752-9431-6501160cac78 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:22:33.717043 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31410c0c-0ba8-41cb-b367-844f46dbf899 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:22:33.717638 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31410c0c-0ba8-41cb-b367-844f46dbf899 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:22:33.717943 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31410c0c-0ba8-41cb-b367-844f46dbf899 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:22:33.718214 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31410c0c-0ba8-41cb-b367-844f46dbf899 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:22:33.719832 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31410c0c-0ba8-41cb-b367-844f46dbf899 None admin] RBAC: Adding query filter params (role.id=a6e562cdc96c4be691d638990f8403fd, scope.project.id=428e1a82439c463f9a1a57d054223644, user.id=0c3f69846e4e4037b7e7165a147c2bde) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:22:33.720133 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31410c0c-0ba8-41cb-b367-844f46dbf899 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:22:33.721547 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31410c0c-0ba8-41cb-b367-844f46dbf899 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:22:33.726793 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 592/1185] 10.222.0.90 () {56 vars in 1374 bytes} [Mon Aug 29 21:22:33 2022] GET /identity/v3/role_assignments?user.id=0c3f69846e4e4037b7e7165a147c2bde&scope.project.id=428e1a82439c463f9a1a57d054223644&role.id=a6e562cdc96c4be691d638990f8403fd => generated 259 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:22:35.580708 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b540a35-dd03-4cc7-a32c-0c0711dc1c07 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:22:35.581278 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b540a35-dd03-4cc7-a32c-0c0711dc1c07 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:22:35.581278 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b540a35-dd03-4cc7-a32c-0c0711dc1c07 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:22:35.582609 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 594/1186] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:22:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:22:35.587821 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-002bd54f-5112-4eae-a7f8-1c56dcb30066 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:22:35.588237 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-002bd54f-5112-4eae-a7f8-1c56dcb30066 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:22:35.588365 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-002bd54f-5112-4eae-a7f8-1c56dcb30066 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:22:35.606397 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-002bd54f-5112-4eae-a7f8-1c56dcb30066 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:22:35.607585 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-002bd54f-5112-4eae-a7f8-1c56dcb30066 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:22:35.672027 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 593/1187] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:22:35 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 21:22:35.938988 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b540a35-dd03-4cc7-a32c-0c0711dc1c07 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:22:35.964659 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:22:36.041587 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b540a35-dd03-4cc7-a32c-0c0711dc1c07 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:22:36.075474 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4ff558a-775e-4be3-abda-2d57fecc1f79 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:22:36.076093 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ff558a-775e-4be3-abda-2d57fecc1f79 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:22:36.076371 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ff558a-775e-4be3-abda-2d57fecc1f79 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:22:36.076640 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ff558a-775e-4be3-abda-2d57fecc1f79 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:22:36.086623 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4ff558a-775e-4be3-abda-2d57fecc1f79 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 21:22:36.091369 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-d4ff558a-775e-4be3-abda-2d57fecc1f79 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 21:22:36.091369 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:22:36.091369 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:22:36.091369 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:22:36.091369 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:22:36.091369 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:22:36.091369 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:22:36.091369 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:22:36.091369 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:22:36.091369 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:22:36.091369 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:22:36.091369 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:22:36.091369 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 21:22:36.091369 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 21:22:36.091369 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 21:22:36.091369 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 21:22:36.092483 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 21:22:36.092483 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 21:22:36.092483 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 21:22:36.092483 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 21:22:36.092483 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:22:36.092483 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:22:36.092483 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:22:36.092483 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:22:36.092483 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:22:36.092483 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:22:36.092483 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:22:36.092483 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:22:36.092483 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:22:36.092483 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:22:36.092483 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:22:36.092483 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:22:36.093362 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:22:36.093362 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:22:36.093362 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:22:36.093362 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:22:36.093362 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 21:22:36.093362 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 21:22:36.093362 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 21:22:36.093362 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 21:22:36.093362 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 21:22:36.093362 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 21:22:36.093362 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 21:22:36.093362 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:22:36.093362 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 595/1188] 10.222.0.90 () {56 vars in 1096 bytes} [Mon Aug 29 21:22:35 2022] GET /identity/v3/roles/admin => generated 83 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:22:36.099216 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-002bd54f-5112-4eae-a7f8-1c56dcb30066 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:22:36.176738 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:22:36.198740 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-002bd54f-5112-4eae-a7f8-1c56dcb30066 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:22:36.232425 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eece992-3b0d-4281-b9f6-9346385e45a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:22:36.233035 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eece992-3b0d-4281-b9f6-9346385e45a4 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:22:36.233318 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eece992-3b0d-4281-b9f6-9346385e45a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:22:36.233592 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eece992-3b0d-4281-b9f6-9346385e45a4 None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:22:36.234823 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eece992-3b0d-4281-b9f6-9346385e45a4 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:22:36.235126 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eece992-3b0d-4281-b9f6-9346385e45a4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:22:36.236492 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eece992-3b0d-4281-b9f6-9346385e45a4 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:22:36.244311 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 594/1189] 10.222.0.90 () {56 vars in 1099 bytes} [Mon Aug 29 21:22:36 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:22:36.250836 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4ff558a-775e-4be3-abda-2d57fecc1f79 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:22:36.335506 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:22:36.352266 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4ff558a-775e-4be3-abda-2d57fecc1f79 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:22:36.385547 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33a769e6-8b17-42d6-bb9c-672bf04f6705 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:22:36.386193 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a769e6-8b17-42d6-bb9c-672bf04f6705 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:22:36.386476 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a769e6-8b17-42d6-bb9c-672bf04f6705 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:22:36.386751 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a769e6-8b17-42d6-bb9c-672bf04f6705 None admin] PATH_INFO: `/v3/users/0c3f69846e4e4037b7e7165a147c2bde` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:22:36.387877 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33a769e6-8b17-42d6-bb9c-672bf04f6705 None admin] RBAC: Authorizing `identity:get_user(user_id=0c3f69846e4e4037b7e7165a147c2bde)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:22:36.389315 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33a769e6-8b17-42d6-bb9c-672bf04f6705 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:22:36.391751 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 596/1190] 10.222.0.90 () {56 vars in 1177 bytes} [Mon Aug 29 21:22:36 2022] GET /identity/v3/users/0c3f69846e4e4037b7e7165a147c2bde => generated 311 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:22:36.397729 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eece992-3b0d-4281-b9f6-9346385e45a4 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:22:36.495225 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eece992-3b0d-4281-b9f6-9346385e45a4 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:22:36.528673 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-215bdce8-b054-4d1c-afae-6eddb506922a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:22:36.529303 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-215bdce8-b054-4d1c-afae-6eddb506922a None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:22:36.529589 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-215bdce8-b054-4d1c-afae-6eddb506922a None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:22:36.529903 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-215bdce8-b054-4d1c-afae-6eddb506922a None admin] PATH_INFO: `/v3/projects/428e1a82439c463f9a1a57d054223644` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:22:36.531123 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-215bdce8-b054-4d1c-afae-6eddb506922a None admin] RBAC: Authorizing `identity:get_project(project_id=428e1a82439c463f9a1a57d054223644)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:22:36.532570 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-215bdce8-b054-4d1c-afae-6eddb506922a None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:22:36.534237 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 595/1191] 10.222.0.90 () {56 vars in 1186 bytes} [Mon Aug 29 21:22:36 2022] GET /identity/v3/projects/428e1a82439c463f9a1a57d054223644 => generated 361 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:22:36.540108 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33a769e6-8b17-42d6-bb9c-672bf04f6705 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:22:36.638811 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33a769e6-8b17-42d6-bb9c-672bf04f6705 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:22:36.672422 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fba7e29-4401-41a1-ae40-eeac4db3e003 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:22:36.673495 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fba7e29-4401-41a1-ae40-eeac4db3e003 None admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:22:36.673751 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fba7e29-4401-41a1-ae40-eeac4db3e003 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:22:36.674108 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fba7e29-4401-41a1-ae40-eeac4db3e003 None admin] PATH_INFO: `/v3/projects/428e1a82439c463f9a1a57d054223644/users/0c3f69846e4e4037b7e7165a147c2bde/roles/a6e562cdc96c4be691d638990f8403fd` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:22:36.676043 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fba7e29-4401-41a1-ae40-eeac4db3e003 None admin] RBAC: Authorizing `identity:create_grant(project_id=428e1a82439c463f9a1a57d054223644, user_id=0c3f69846e4e4037b7e7165a147c2bde, role_id=a6e562cdc96c4be691d638990f8403fd)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:22:36.677470 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fba7e29-4401-41a1-ae40-eeac4db3e003 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:22:36.691479 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 597/1192] 10.222.0.90 () {58 vars in 1439 bytes} [Mon Aug 29 21:22:36 2022] PUT /identity/v3/projects/428e1a82439c463f9a1a57d054223644/users/0c3f69846e4e4037b7e7165a147c2bde/roles/a6e562cdc96c4be691d638990f8403fd => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:22:38.583049 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3941ceb4-ff81-499b-a727-df905ab4f20b None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:22:38.583589 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3941ceb4-ff81-499b-a727-df905ab4f20b None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:22:38.583589 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3941ceb4-ff81-499b-a727-df905ab4f20b None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:22:38.584904 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 596/1193] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:22:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:22:38.589985 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1cc942d-5cf1-4587-b188-bb3bef52b915 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:22:38.590378 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1cc942d-5cf1-4587-b188-bb3bef52b915 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:22:38.590551 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1cc942d-5cf1-4587-b188-bb3bef52b915 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:22:38.609860 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-e1cc942d-5cf1-4587-b188-bb3bef52b915 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:22:38.611195 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-e1cc942d-5cf1-4587-b188-bb3bef52b915 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:22:38.618309 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:22:38.680064 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 598/1194] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:22:38 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 21:22:38.946597 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3941ceb4-ff81-499b-a727-df905ab4f20b None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:22:38.958625 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:22:39.049189 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3941ceb4-ff81-499b-a727-df905ab4f20b None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:22:39.082031 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-586dfdba-21b4-432b-b9b6-a0478d826cf3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:22:39.082665 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-586dfdba-21b4-432b-b9b6-a0478d826cf3 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:22:39.082949 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-586dfdba-21b4-432b-b9b6-a0478d826cf3 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:22:39.083226 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-586dfdba-21b4-432b-b9b6-a0478d826cf3 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:22:39.095049 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-586dfdba-21b4-432b-b9b6-a0478d826cf3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 21:22:39.099861 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-586dfdba-21b4-432b-b9b6-a0478d826cf3 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 21:22:39.099861 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:22:39.099861 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:22:39.099861 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:22:39.099861 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:22:39.099861 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:22:39.099861 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:22:39.099861 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:22:39.099861 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:22:39.099861 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:22:39.099861 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:22:39.099861 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:22:39.099861 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 21:22:39.099861 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 21:22:39.099861 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 21:22:39.099861 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 21:22:39.101009 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 21:22:39.101009 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 21:22:39.101009 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 21:22:39.101009 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 21:22:39.101009 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:22:39.101009 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:22:39.101009 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:22:39.101009 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:22:39.101009 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:22:39.101009 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:22:39.101009 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:22:39.101009 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:22:39.101009 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:22:39.101009 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:22:39.101009 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:22:39.101009 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:22:39.102157 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:22:39.102157 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:22:39.102157 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:22:39.102157 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:22:39.102157 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 21:22:39.102157 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 21:22:39.102157 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 21:22:39.102157 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 21:22:39.102157 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 21:22:39.102157 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 21:22:39.102157 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 21:22:39.102157 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:22:39.102157 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 597/1195] 10.222.0.90 () {56 vars in 1096 bytes} [Mon Aug 29 21:22:38 2022] GET /identity/v3/roles/admin => generated 83 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:22:39.107163 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1cc942d-5cf1-4587-b188-bb3bef52b915 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:22:39.204152 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1cc942d-5cf1-4587-b188-bb3bef52b915 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:22:39.238550 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceb4f29f-3120-446f-974d-7a53a641feb9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:22:39.239147 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceb4f29f-3120-446f-974d-7a53a641feb9 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:22:39.239428 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceb4f29f-3120-446f-974d-7a53a641feb9 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:22:39.239700 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceb4f29f-3120-446f-974d-7a53a641feb9 None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:22:39.240363 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceb4f29f-3120-446f-974d-7a53a641feb9 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:22:39.240666 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceb4f29f-3120-446f-974d-7a53a641feb9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:22:39.242048 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceb4f29f-3120-446f-974d-7a53a641feb9 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:22:39.249919 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 599/1196] 10.222.0.90 () {56 vars in 1099 bytes} [Mon Aug 29 21:22:39 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:22:39.255753 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-586dfdba-21b4-432b-b9b6-a0478d826cf3 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:22:39.320813 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:22:39.354367 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-586dfdba-21b4-432b-b9b6-a0478d826cf3 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:22:39.387540 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbc60998-e4e8-4e5a-b56d-d21f640eea8b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:22:39.388167 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbc60998-e4e8-4e5a-b56d-d21f640eea8b None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:22:39.388448 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbc60998-e4e8-4e5a-b56d-d21f640eea8b None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:22:39.388724 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbc60998-e4e8-4e5a-b56d-d21f640eea8b None admin] PATH_INFO: `/v3/users/0c3f69846e4e4037b7e7165a147c2bde` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:22:39.389872 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbc60998-e4e8-4e5a-b56d-d21f640eea8b None admin] RBAC: Authorizing `identity:get_user(user_id=0c3f69846e4e4037b7e7165a147c2bde)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:22:39.391384 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbc60998-e4e8-4e5a-b56d-d21f640eea8b None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:22:39.392949 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 598/1197] 10.222.0.90 () {56 vars in 1177 bytes} [Mon Aug 29 21:22:39 2022] GET /identity/v3/users/0c3f69846e4e4037b7e7165a147c2bde => generated 311 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:22:39.398985 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceb4f29f-3120-446f-974d-7a53a641feb9 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:22:39.402726 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:22:39.498605 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceb4f29f-3120-446f-974d-7a53a641feb9 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:22:39.530632 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11f43698-8b9b-4e93-b596-a326ed763321 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:22:39.531849 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11f43698-8b9b-4e93-b596-a326ed763321 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:22:39.532122 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11f43698-8b9b-4e93-b596-a326ed763321 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:22:39.532390 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11f43698-8b9b-4e93-b596-a326ed763321 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:22:39.598068 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11f43698-8b9b-4e93-b596-a326ed763321 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:22:39.599504 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11f43698-8b9b-4e93-b596-a326ed763321 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:22:39.602035 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:22:39.695271 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 600/1198] 10.222.0.90 () {58 vars in 1282 bytes} [Mon Aug 29 21:22:39 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 297 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 29 21:22:39.701395 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbc60998-e4e8-4e5a-b56d-d21f640eea8b None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:22:39.805369 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbc60998-e4e8-4e5a-b56d-d21f640eea8b None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:22:39.838518 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12504c2f-5a2f-4861-b86a-3079f32c1b32 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:22:39.839143 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12504c2f-5a2f-4861-b86a-3079f32c1b32 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:22:39.839424 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12504c2f-5a2f-4861-b86a-3079f32c1b32 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:22:39.839697 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12504c2f-5a2f-4861-b86a-3079f32c1b32 None admin] PATH_INFO: `/v3/projects/428e1a82439c463f9a1a57d054223644` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:22:39.840814 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12504c2f-5a2f-4861-b86a-3079f32c1b32 None admin] RBAC: Authorizing `identity:get_project(project_id=428e1a82439c463f9a1a57d054223644)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:22:39.842285 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12504c2f-5a2f-4861-b86a-3079f32c1b32 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:22:39.845039 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 599/1199] 10.222.0.90 () {56 vars in 1186 bytes} [Mon Aug 29 21:22:39 2022] GET /identity/v3/projects/428e1a82439c463f9a1a57d054223644 => generated 361 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:22:39.851388 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11f43698-8b9b-4e93-b596-a326ed763321 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:22:39.947826 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11f43698-8b9b-4e93-b596-a326ed763321 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:22:39.950135 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:22:39.982793 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a89f1757-bd21-4ba0-9d37-5fcee5fcc575 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:22:39.983395 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a89f1757-bd21-4ba0-9d37-5fcee5fcc575 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:22:39.983673 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a89f1757-bd21-4ba0-9d37-5fcee5fcc575 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:22:39.984030 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a89f1757-bd21-4ba0-9d37-5fcee5fcc575 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:22:39.984758 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a89f1757-bd21-4ba0-9d37-5fcee5fcc575 None admin] RBAC: Adding query filter params (role.id=a6e562cdc96c4be691d638990f8403fd, scope.project.id=428e1a82439c463f9a1a57d054223644, user.id=0c3f69846e4e4037b7e7165a147c2bde) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:22:39.985061 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a89f1757-bd21-4ba0-9d37-5fcee5fcc575 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:22:39.986506 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a89f1757-bd21-4ba0-9d37-5fcee5fcc575 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:22:39.992137 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 601/1200] 10.222.0.90 () {56 vars in 1374 bytes} [Mon Aug 29 21:22:39 2022] GET /identity/v3/role_assignments?user.id=0c3f69846e4e4037b7e7165a147c2bde&scope.project.id=428e1a82439c463f9a1a57d054223644&role.id=a6e562cdc96c4be691d638990f8403fd => generated 608 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:22:49.834278 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=98 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:22:49.989567 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=16 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:23:35.804571 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93048749-cc95-4090-9782-cca1e07be922 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:23:35.805176 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93048749-cc95-4090-9782-cca1e07be922 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:23:35.805176 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93048749-cc95-4090-9782-cca1e07be922 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:23:35.806569 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 600/1201] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:23:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:23:35.812840 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eed793b-c123-4382-b7c6-d309950e40eb None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:23:35.813385 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eed793b-c123-4382-b7c6-d309950e40eb None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:23:35.813385 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eed793b-c123-4382-b7c6-d309950e40eb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:23:35.834917 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-9eed793b-c123-4382-b7c6-d309950e40eb None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:23:35.836355 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-9eed793b-c123-4382-b7c6-d309950e40eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:23:35.938164 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 602/1202] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:23:35 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 21:23:36.226129 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93048749-cc95-4090-9782-cca1e07be922 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:23:36.378363 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93048749-cc95-4090-9782-cca1e07be922 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:23:36.424439 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84df5e85-18c1-4bba-b1da-364f61ef6ade None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:23:36.425277 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84df5e85-18c1-4bba-b1da-364f61ef6ade None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:23:36.425612 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84df5e85-18c1-4bba-b1da-364f61ef6ade None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:23:36.425992 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84df5e85-18c1-4bba-b1da-364f61ef6ade None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:23:36.439494 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84df5e85-18c1-4bba-b1da-364f61ef6ade None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:23:36.441944 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84df5e85-18c1-4bba-b1da-364f61ef6ade None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:23:36.460882 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-84df5e85-18c1-4bba-b1da-364f61ef6ade None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:23:36.460882 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:23:36.460882 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:23:36.460882 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:23:36.460882 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:23:36.460882 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:23:36.460882 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:23:36.460882 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:23:36.460882 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:23:36.460882 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:23:36.460882 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:23:36.460882 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:23:36.460882 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 21:23:36.460882 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 21:23:36.460882 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 21:23:36.460882 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 21:23:36.462172 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:23:36.462172 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:23:36.462172 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:23:36.462172 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:23:36.462172 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:23:36.462172 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:23:36.462172 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:23:36.462172 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:23:36.462172 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:23:36.462172 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:23:36.462172 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:23:36.462172 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:23:36.462172 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:23:36.462172 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:23:36.462172 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:23:36.462172 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:23:36.462172 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 21:23:36.463365 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 21:23:36.463365 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:23:36.463365 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:23:36.466162 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 601/1203] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:23:36 2022] GET /identity/v3/domains/Default => generated 87 bytes in 241 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:23:36.473581 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eed793b-c123-4382-b7c6-d309950e40eb None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:23:36.614018 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eed793b-c123-4382-b7c6-d309950e40eb None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:23:36.659265 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:23:36.667297 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-154d81d8-f59f-4136-8f31-51ad144a75d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:23:36.668839 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-154d81d8-f59f-4136-8f31-51ad144a75d1 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:23:36.669367 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-154d81d8-f59f-4136-8f31-51ad144a75d1 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:23:36.670134 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-154d81d8-f59f-4136-8f31-51ad144a75d1 None admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:23:36.671039 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-154d81d8-f59f-4136-8f31-51ad144a75d1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:23:36.671573 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-154d81d8-f59f-4136-8f31-51ad144a75d1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:23:36.673314 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-154d81d8-f59f-4136-8f31-51ad144a75d1 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:23:36.692151 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 603/1204] 10.222.0.90 () {56 vars in 1109 bytes} [Mon Aug 29 21:23:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:23:36.698818 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84df5e85-18c1-4bba-b1da-364f61ef6ade None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:23:36.788327 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:23:36.863050 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84df5e85-18c1-4bba-b1da-364f61ef6ade None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:23:36.910856 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd21b746-066a-42c0-98ff-b48be7aa7dd2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:23:36.911452 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd21b746-066a-42c0-98ff-b48be7aa7dd2 None admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:23:36.911688 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd21b746-066a-42c0-98ff-b48be7aa7dd2 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:23:36.912111 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd21b746-066a-42c0-98ff-b48be7aa7dd2 None admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:23:36.912965 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd21b746-066a-42c0-98ff-b48be7aa7dd2 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:23:36.914494 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd21b746-066a-42c0-98ff-b48be7aa7dd2 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:23:36.949824 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 602/1205] 10.222.0.90 () {60 vars in 1132 bytes} [Mon Aug 29 21:23:36 2022] POST /identity/v3/users => generated 251 bytes in 245 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:23:39.238925 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e224b3d6-e2cb-46bb-8ba7-da2cb0842ce1 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:23:39.238925 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e224b3d6-e2cb-46bb-8ba7-da2cb0842ce1 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:23:39.238925 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e224b3d6-e2cb-46bb-8ba7-da2cb0842ce1 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:23:39.239724 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 604/1206] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:23:39 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:23:39.245850 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99dcac8d-2df9-49d1-9e90-3dc54a5bed58 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:23:39.246599 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99dcac8d-2df9-49d1-9e90-3dc54a5bed58 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:23:39.246599 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99dcac8d-2df9-49d1-9e90-3dc54a5bed58 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:23:39.268741 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-99dcac8d-2df9-49d1-9e90-3dc54a5bed58 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:23:39.269982 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-99dcac8d-2df9-49d1-9e90-3dc54a5bed58 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:23:39.360340 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 603/1207] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:23:39 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 21:23:39.661637 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e224b3d6-e2cb-46bb-8ba7-da2cb0842ce1 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:23:39.791932 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e224b3d6-e2cb-46bb-8ba7-da2cb0842ce1 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:23:39.837995 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfe81889-a2c6-4c40-ba90-2c431e13fc95 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:23:39.838939 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfe81889-a2c6-4c40-ba90-2c431e13fc95 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:23:39.839333 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfe81889-a2c6-4c40-ba90-2c431e13fc95 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:23:39.839766 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfe81889-a2c6-4c40-ba90-2c431e13fc95 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:23:39.864738 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfe81889-a2c6-4c40-ba90-2c431e13fc95 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 21:23:39.871350 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-cfe81889-a2c6-4c40-ba90-2c431e13fc95 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 29 21:23:39.871350 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:23:39.871350 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:23:39.871350 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:23:39.871350 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:23:39.871350 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:23:39.871350 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:23:39.871350 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:23:39.871350 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:23:39.871350 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:23:39.871350 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:23:39.871350 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:23:39.871350 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 21:23:39.871350 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 21:23:39.871350 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 21:23:39.871350 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 21:23:39.872509 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 21:23:39.872509 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 21:23:39.872509 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 21:23:39.872509 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 21:23:39.872509 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:23:39.872509 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:23:39.872509 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:23:39.872509 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:23:39.872509 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:23:39.872509 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:23:39.872509 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:23:39.872509 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:23:39.872509 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:23:39.872509 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:23:39.872509 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:23:39.872509 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:23:39.873912 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:23:39.873912 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:23:39.873912 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:23:39.873912 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:23:39.873912 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 21:23:39.873912 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 21:23:39.873912 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 21:23:39.873912 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 21:23:39.873912 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 21:23:39.873912 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 21:23:39.873912 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 29 21:23:39.873912 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:23:39.873912 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 605/1208] 10.222.0.90 () {56 vars in 1102 bytes} [Mon Aug 29 21:23:39 2022] GET /identity/v3/roles/service => generated 85 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:23:39.878714 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99dcac8d-2df9-49d1-9e90-3dc54a5bed58 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:23:39.929081 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:23:40.000986 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99dcac8d-2df9-49d1-9e90-3dc54a5bed58 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:23:40.040220 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fef79a83-f279-45fe-969c-7bd43ac9b843 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:23:40.041184 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fef79a83-f279-45fe-969c-7bd43ac9b843 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:23:40.041690 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fef79a83-f279-45fe-969c-7bd43ac9b843 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:23:40.041992 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fef79a83-f279-45fe-969c-7bd43ac9b843 None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:23:40.042843 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fef79a83-f279-45fe-969c-7bd43ac9b843 None admin] RBAC: Adding query filter params (name=service) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:23:40.043205 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fef79a83-f279-45fe-969c-7bd43ac9b843 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:23:40.044794 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fef79a83-f279-45fe-969c-7bd43ac9b843 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:23:40.061363 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 604/1209] 10.222.0.90 () {56 vars in 1103 bytes} [Mon Aug 29 21:23:39 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:23:40.063052 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfe81889-a2c6-4c40-ba90-2c431e13fc95 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:23:40.141006 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:23:40.206916 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfe81889-a2c6-4c40-ba90-2c431e13fc95 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:23:40.251491 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6804c1e-9b3b-4839-898d-ce26473226a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:23:40.252452 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6804c1e-9b3b-4839-898d-ce26473226a5 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:23:40.252665 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6804c1e-9b3b-4839-898d-ce26473226a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:23:40.252953 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6804c1e-9b3b-4839-898d-ce26473226a5 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:23:40.269299 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6804c1e-9b3b-4839-898d-ce26473226a5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:23:40.271650 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6804c1e-9b3b-4839-898d-ce26473226a5 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:23:40.294388 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-b6804c1e-9b3b-4839-898d-ce26473226a5 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:23:40.294388 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:23:40.294388 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:23:40.294388 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:23:40.294388 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:23:40.294388 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:23:40.294388 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:23:40.294388 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:23:40.294388 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:23:40.294388 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:23:40.294388 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:23:40.294388 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:23:40.294388 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 21:23:40.294388 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 21:23:40.294388 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 21:23:40.294388 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 21:23:40.295615 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:23:40.295615 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:23:40.295615 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:23:40.295615 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:23:40.295615 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:23:40.295615 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:23:40.295615 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:23:40.295615 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:23:40.295615 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:23:40.295615 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:23:40.295615 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:23:40.295615 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:23:40.295615 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:23:40.295615 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:23:40.295615 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:23:40.295615 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:23:40.295615 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 21:23:40.296540 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 21:23:40.296540 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:23:40.296540 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:23:40.296540 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 606/1210] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:23:40 2022] GET /identity/v3/domains/Default => generated 87 bytes in 233 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:23:40.304003 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fef79a83-f279-45fe-969c-7bd43ac9b843 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:23:40.421675 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:23:40.424910 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fef79a83-f279-45fe-969c-7bd43ac9b843 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:23:40.471118 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c37bf47d-e65a-49eb-80e0-bab5327198b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:23:40.471118 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c37bf47d-e65a-49eb-80e0-bab5327198b3 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:23:40.471118 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c37bf47d-e65a-49eb-80e0-bab5327198b3 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:23:40.471466 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c37bf47d-e65a-49eb-80e0-bab5327198b3 None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:23:40.471999 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c37bf47d-e65a-49eb-80e0-bab5327198b3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:23:40.472367 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c37bf47d-e65a-49eb-80e0-bab5327198b3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:23:40.474019 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c37bf47d-e65a-49eb-80e0-bab5327198b3 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:23:40.489976 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 605/1211] 10.222.0.90 () {56 vars in 1109 bytes} [Mon Aug 29 21:23:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:23:40.496470 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6804c1e-9b3b-4839-898d-ce26473226a5 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:23:40.623660 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6804c1e-9b3b-4839-898d-ce26473226a5 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:23:40.625689 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:23:40.669604 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd7b75d1-9fd3-46c2-948a-70a7528a2837 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:23:40.670311 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd7b75d1-9fd3-46c2-948a-70a7528a2837 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:23:40.670584 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd7b75d1-9fd3-46c2-948a-70a7528a2837 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:23:40.670891 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd7b75d1-9fd3-46c2-948a-70a7528a2837 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:23:40.680090 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd7b75d1-9fd3-46c2-948a-70a7528a2837 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:23:40.681958 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd7b75d1-9fd3-46c2-948a-70a7528a2837 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:23:40.698690 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-dd7b75d1-9fd3-46c2-948a-70a7528a2837 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Aug 29 21:23:40.698690 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:23:40.698690 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:23:40.698690 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:23:40.698690 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:23:40.698690 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:23:40.698690 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:23:40.698690 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:23:40.698690 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:23:40.698690 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:23:40.698690 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:23:40.698690 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:23:40.698690 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 29 21:23:40.698690 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 29 21:23:40.698690 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 29 21:23:40.698690 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 29 21:23:40.701234 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:23:40.701234 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:23:40.701234 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 29 21:23:40.701234 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 21:23:40.701234 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 29 21:23:40.701234 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 21:23:40.701234 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:23:40.701234 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:23:40.701234 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:23:40.701234 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:23:40.701234 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:23:40.701234 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:23:40.701234 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:23:40.701234 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:23:40.701234 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:23:40.701234 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:23:40.701234 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:23:40.701234 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:23:40.702264 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:23:40.702264 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:23:40.702264 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 29 21:23:40.702264 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 29 21:23:40.702264 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 29 21:23:40.702264 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 29 21:23:40.702264 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 29 21:23:40.702264 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 29 21:23:40.702264 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Aug 29 21:23:40.702264 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:23:40.702264 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 607/1212] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:23:40 2022] GET /identity/v3/users/placement => generated 87 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:23:40.707567 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c37bf47d-e65a-49eb-80e0-bab5327198b3 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:23:40.837035 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c37bf47d-e65a-49eb-80e0-bab5327198b3 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:23:40.882294 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-800e16d9-d235-45d7-9994-a96a44db8e80 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:23:40.882951 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-800e16d9-d235-45d7-9994-a96a44db8e80 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:23:40.883227 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-800e16d9-d235-45d7-9994-a96a44db8e80 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:23:40.883520 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-800e16d9-d235-45d7-9994-a96a44db8e80 None admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:23:40.884295 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-800e16d9-d235-45d7-9994-a96a44db8e80 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:23:40.884623 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-800e16d9-d235-45d7-9994-a96a44db8e80 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:23:40.886164 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-800e16d9-d235-45d7-9994-a96a44db8e80 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:23:40.912798 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 606/1213] 10.222.0.90 () {56 vars in 1143 bytes} [Mon Aug 29 21:23:40 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:23:40.914643 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd7b75d1-9fd3-46c2-948a-70a7528a2837 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:23:41.046402 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd7b75d1-9fd3-46c2-948a-70a7528a2837 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:23:41.083788 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ebbe773-d16b-4433-9513-e9085d14f7a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:23:41.084405 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ebbe773-d16b-4433-9513-e9085d14f7a4 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:23:41.084680 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ebbe773-d16b-4433-9513-e9085d14f7a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:23:41.084953 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ebbe773-d16b-4433-9513-e9085d14f7a4 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:23:41.120551 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:23:41.162503 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ebbe773-d16b-4433-9513-e9085d14f7a4 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:23:41.164414 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ebbe773-d16b-4433-9513-e9085d14f7a4 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:23:41.282869 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 608/1214] 10.222.0.90 () {58 vars in 1282 bytes} [Mon Aug 29 21:23:40 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 364 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Aug 29 21:23:41.284750 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-800e16d9-d235-45d7-9994-a96a44db8e80 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:23:41.304686 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:23:41.397633 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-800e16d9-d235-45d7-9994-a96a44db8e80 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:23:41.433328 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57397267-e905-4795-a0cf-e24e6b10119b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:23:41.434124 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57397267-e905-4795-a0cf-e24e6b10119b None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:23:41.434339 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57397267-e905-4795-a0cf-e24e6b10119b None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:23:41.434516 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57397267-e905-4795-a0cf-e24e6b10119b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:23:41.445054 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57397267-e905-4795-a0cf-e24e6b10119b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:23:41.446536 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57397267-e905-4795-a0cf-e24e6b10119b None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:23:41.463896 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-57397267-e905-4795-a0cf-e24e6b10119b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:23:41.463896 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:23:41.463896 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:23:41.463896 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:23:41.463896 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:23:41.463896 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:23:41.463896 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:23:41.463896 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:23:41.463896 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:23:41.463896 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:23:41.463896 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:23:41.463896 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:23:41.463896 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 21:23:41.463896 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 21:23:41.463896 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 21:23:41.463896 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 21:23:41.466582 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:23:41.466582 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:23:41.466582 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:23:41.466582 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:23:41.466582 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:23:41.466582 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:23:41.466582 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:23:41.466582 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:23:41.466582 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:23:41.466582 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:23:41.466582 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:23:41.466582 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:23:41.466582 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:23:41.466582 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:23:41.466582 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:23:41.466582 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:23:41.466582 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 21:23:41.467980 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 21:23:41.467980 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:23:41.467980 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:23:41.467980 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 607/1215] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:23:41 2022] GET /identity/v3/domains/Default => generated 87 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 29 21:23:41.474995 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ebbe773-d16b-4433-9513-e9085d14f7a4 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:23:41.545423 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:23:41.591879 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ebbe773-d16b-4433-9513-e9085d14f7a4 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:23:41.633110 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe31f3b3-cbf6-4018-a8c5-d736a8c4462b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:23:41.633847 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe31f3b3-cbf6-4018-a8c5-d736a8c4462b None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:23:41.634198 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe31f3b3-cbf6-4018-a8c5-d736a8c4462b None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:23:41.634538 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe31f3b3-cbf6-4018-a8c5-d736a8c4462b None admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:23:41.635250 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe31f3b3-cbf6-4018-a8c5-d736a8c4462b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:23:41.635620 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe31f3b3-cbf6-4018-a8c5-d736a8c4462b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:23:41.637664 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe31f3b3-cbf6-4018-a8c5-d736a8c4462b None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:23:41.650318 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 609/1216] 10.222.0.90 () {56 vars in 1109 bytes} [Mon Aug 29 21:23:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:23:41.656044 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57397267-e905-4795-a0cf-e24e6b10119b None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:23:41.743979 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:23:41.782615 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57397267-e905-4795-a0cf-e24e6b10119b None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:23:41.824836 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef0e58b3-253a-481b-8a5a-9e5fa6177eac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:23:41.825858 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef0e58b3-253a-481b-8a5a-9e5fa6177eac None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:23:41.826302 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef0e58b3-253a-481b-8a5a-9e5fa6177eac None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:23:41.826688 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef0e58b3-253a-481b-8a5a-9e5fa6177eac None admin] PATH_INFO: `/v3/projects/service` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:23:41.834150 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef0e58b3-253a-481b-8a5a-9e5fa6177eac None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:23:41.835601 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef0e58b3-253a-481b-8a5a-9e5fa6177eac None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:23:41.852227 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-ef0e58b3-253a-481b-8a5a-9e5fa6177eac None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 21:23:41.852227 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:23:41.852227 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:23:41.852227 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:23:41.852227 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:23:41.852227 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:23:41.852227 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:23:41.852227 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:23:41.852227 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:23:41.852227 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:23:41.852227 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:23:41.852227 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:23:41.852227 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 29 21:23:41.852227 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 29 21:23:41.852227 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 29 21:23:41.853434 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 29 21:23:41.853434 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:23:41.853434 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:23:41.853434 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:23:41.853434 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:23:41.853434 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:23:41.853434 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:23:41.853434 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:23:41.853434 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:23:41.853434 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:23:41.853434 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:23:41.853434 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:23:41.853434 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:23:41.853434 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:23:41.853434 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:23:41.853434 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:23:41.853434 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:23:41.862399 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 29 21:23:41.862399 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 29 21:23:41.862399 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 29 21:23:41.862399 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 29 21:23:41.862399 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 29 21:23:41.862399 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 29 21:23:41.862399 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 21:23:41.862399 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:23:41.862399 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 608/1217] 10.222.0.90 () {56 vars in 1111 bytes} [Mon Aug 29 21:23:41 2022] GET /identity/v3/projects/service => generated 88 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 29 21:23:41.862953 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe31f3b3-cbf6-4018-a8c5-d736a8c4462b None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:23:41.986410 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe31f3b3-cbf6-4018-a8c5-d736a8c4462b None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:23:42.019834 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:23:42.038489 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16095cc3-f2f7-4f6f-8064-4cdea64b8f99 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:23:42.039204 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16095cc3-f2f7-4f6f-8064-4cdea64b8f99 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:23:42.039494 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16095cc3-f2f7-4f6f-8064-4cdea64b8f99 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:23:42.039821 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16095cc3-f2f7-4f6f-8064-4cdea64b8f99 None admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:23:42.040846 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16095cc3-f2f7-4f6f-8064-4cdea64b8f99 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:23:42.041362 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16095cc3-f2f7-4f6f-8064-4cdea64b8f99 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:23:42.043363 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16095cc3-f2f7-4f6f-8064-4cdea64b8f99 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:23:42.061435 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 610/1218] 10.222.0.90 () {56 vars in 1148 bytes} [Mon Aug 29 21:23:41 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:23:42.062630 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef0e58b3-253a-481b-8a5a-9e5fa6177eac None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:23:42.200840 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef0e58b3-253a-481b-8a5a-9e5fa6177eac None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:23:42.236103 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:23:42.246341 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18123959-7975-4369-a998-9b5a7f3a92b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:23:42.246925 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18123959-7975-4369-a998-9b5a7f3a92b5 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:23:42.247354 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18123959-7975-4369-a998-9b5a7f3a92b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:23:42.247574 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18123959-7975-4369-a998-9b5a7f3a92b5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:23:42.248568 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18123959-7975-4369-a998-9b5a7f3a92b5 None admin] RBAC: Adding query filter params (role.id=a94bdb4c5796428baf63644c062dfcb2, scope.project.id=ec03e112e3d547fbab6faf7b485c3fca, user.id=b95fa07a7c0549dbaac49406e2639e21) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:23:42.248920 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18123959-7975-4369-a998-9b5a7f3a92b5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:23:42.250786 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18123959-7975-4369-a998-9b5a7f3a92b5 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:23:42.258340 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 609/1219] 10.222.0.90 () {56 vars in 1374 bytes} [Mon Aug 29 21:23:42 2022] GET /identity/v3/role_assignments?user.id=b95fa07a7c0549dbaac49406e2639e21&scope.project.id=ec03e112e3d547fbab6faf7b485c3fca&role.id=a94bdb4c5796428baf63644c062dfcb2 => generated 259 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:23:44.506387 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f7b74fa-c7ae-44da-a78d-cf2f166ad970 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:23:44.506972 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f7b74fa-c7ae-44da-a78d-cf2f166ad970 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:23:44.506972 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f7b74fa-c7ae-44da-a78d-cf2f166ad970 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:23:44.508286 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 611/1220] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:23:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:23:44.514736 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd87561c-1bc0-470a-b2ea-8d63746204fb None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:23:44.515343 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd87561c-1bc0-470a-b2ea-8d63746204fb None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:23:44.515343 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd87561c-1bc0-470a-b2ea-8d63746204fb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:23:44.533759 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-dd87561c-1bc0-470a-b2ea-8d63746204fb None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:23:44.534996 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-dd87561c-1bc0-470a-b2ea-8d63746204fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:23:44.606417 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 610/1221] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:23:44 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 21:23:44.885538 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f7b74fa-c7ae-44da-a78d-cf2f166ad970 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:23:45.009654 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f7b74fa-c7ae-44da-a78d-cf2f166ad970 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:23:45.051330 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f76f111-bd46-49eb-b30f-787d8d22ef87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:23:45.051971 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f76f111-bd46-49eb-b30f-787d8d22ef87 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:23:45.052251 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f76f111-bd46-49eb-b30f-787d8d22ef87 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:23:45.052512 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f76f111-bd46-49eb-b30f-787d8d22ef87 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:23:45.063393 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f76f111-bd46-49eb-b30f-787d8d22ef87 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 21:23:45.068147 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-6f76f111-bd46-49eb-b30f-787d8d22ef87 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 29 21:23:45.068147 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:23:45.068147 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:23:45.068147 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:23:45.068147 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:23:45.068147 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:23:45.068147 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:23:45.068147 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:23:45.068147 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:23:45.068147 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:23:45.068147 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:23:45.068147 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:23:45.068147 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 21:23:45.068147 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 21:23:45.068147 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 21:23:45.068147 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 21:23:45.069251 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 21:23:45.069251 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 21:23:45.069251 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 21:23:45.069251 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 21:23:45.069251 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:23:45.069251 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:23:45.069251 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:23:45.069251 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:23:45.069251 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:23:45.069251 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:23:45.069251 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:23:45.069251 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:23:45.069251 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:23:45.069251 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:23:45.069251 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:23:45.069251 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:23:45.070175 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:23:45.070175 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:23:45.070175 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:23:45.070175 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:23:45.070175 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 21:23:45.070175 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 21:23:45.070175 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 21:23:45.070175 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 21:23:45.070175 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 21:23:45.070175 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 21:23:45.070175 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 29 21:23:45.070175 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:23:45.070175 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 612/1222] 10.222.0.90 () {56 vars in 1102 bytes} [Mon Aug 29 21:23:44 2022] GET /identity/v3/roles/service => generated 85 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:23:45.076001 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd87561c-1bc0-470a-b2ea-8d63746204fb None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:23:45.186337 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd87561c-1bc0-470a-b2ea-8d63746204fb None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:23:45.213205 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:23:45.226628 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bcf5272-98a7-4c34-a7f0-f636a6ad0792 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:23:45.227283 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bcf5272-98a7-4c34-a7f0-f636a6ad0792 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:23:45.227578 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bcf5272-98a7-4c34-a7f0-f636a6ad0792 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:23:45.227857 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bcf5272-98a7-4c34-a7f0-f636a6ad0792 None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:23:45.228568 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bcf5272-98a7-4c34-a7f0-f636a6ad0792 None admin] RBAC: Adding query filter params (name=service) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:23:45.228927 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bcf5272-98a7-4c34-a7f0-f636a6ad0792 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:23:45.231184 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bcf5272-98a7-4c34-a7f0-f636a6ad0792 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:23:45.239987 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 611/1223] 10.222.0.90 () {56 vars in 1103 bytes} [Mon Aug 29 21:23:45 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:23:45.246942 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f76f111-bd46-49eb-b30f-787d8d22ef87 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:23:45.300860 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:23:45.386480 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f76f111-bd46-49eb-b30f-787d8d22ef87 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:23:45.426219 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28db563c-ce25-4e65-b45d-76d489bfc568 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:23:45.426949 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28db563c-ce25-4e65-b45d-76d489bfc568 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:23:45.427229 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28db563c-ce25-4e65-b45d-76d489bfc568 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:23:45.427501 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28db563c-ce25-4e65-b45d-76d489bfc568 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:23:45.439624 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28db563c-ce25-4e65-b45d-76d489bfc568 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:23:45.441120 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28db563c-ce25-4e65-b45d-76d489bfc568 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:23:45.459418 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-28db563c-ce25-4e65-b45d-76d489bfc568 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:23:45.459418 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:23:45.459418 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:23:45.459418 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:23:45.459418 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:23:45.459418 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:23:45.459418 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:23:45.459418 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:23:45.459418 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:23:45.459418 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:23:45.459418 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:23:45.459418 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:23:45.459418 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 21:23:45.459418 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 21:23:45.459418 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 21:23:45.459418 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 21:23:45.460748 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:23:45.460748 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:23:45.460748 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:23:45.460748 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:23:45.460748 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:23:45.460748 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:23:45.460748 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:23:45.460748 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:23:45.460748 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:23:45.460748 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:23:45.460748 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:23:45.460748 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:23:45.460748 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:23:45.460748 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:23:45.460748 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:23:45.460748 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:23:45.460748 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 21:23:45.462409 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 21:23:45.462409 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:23:45.462409 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:23:45.462409 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 613/1224] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:23:45 2022] GET /identity/v3/domains/Default => generated 87 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:23:45.468158 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bcf5272-98a7-4c34-a7f0-f636a6ad0792 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:23:45.586282 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bcf5272-98a7-4c34-a7f0-f636a6ad0792 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:23:45.624071 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d871fc0a-9c27-458d-ad52-c301cd22039e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:23:45.624758 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d871fc0a-9c27-458d-ad52-c301cd22039e None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:23:45.625064 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d871fc0a-9c27-458d-ad52-c301cd22039e None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:23:45.625343 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d871fc0a-9c27-458d-ad52-c301cd22039e None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:23:45.626096 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d871fc0a-9c27-458d-ad52-c301cd22039e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:23:45.626381 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d871fc0a-9c27-458d-ad52-c301cd22039e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:23:45.627845 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d871fc0a-9c27-458d-ad52-c301cd22039e None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:23:45.642825 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 612/1225] 10.222.0.90 () {56 vars in 1109 bytes} [Mon Aug 29 21:23:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:23:45.648671 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28db563c-ce25-4e65-b45d-76d489bfc568 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:23:45.754068 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:23:45.763095 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28db563c-ce25-4e65-b45d-76d489bfc568 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:23:45.800992 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a49ad93-dd22-4c97-8b5a-597d8c81b15c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:23:45.801662 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a49ad93-dd22-4c97-8b5a-597d8c81b15c None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:23:45.802011 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a49ad93-dd22-4c97-8b5a-597d8c81b15c None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:23:45.802332 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a49ad93-dd22-4c97-8b5a-597d8c81b15c None admin] PATH_INFO: `/v3/users/placement` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:23:45.811027 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a49ad93-dd22-4c97-8b5a-597d8c81b15c None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:23:45.812597 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a49ad93-dd22-4c97-8b5a-597d8c81b15c None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:23:45.825484 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-0a49ad93-dd22-4c97-8b5a-597d8c81b15c None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Aug 29 21:23:45.825484 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:23:45.825484 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:23:45.825484 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:23:45.825484 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:23:45.825484 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:23:45.825484 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:23:45.825484 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:23:45.825484 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:23:45.825484 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:23:45.825484 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:23:45.825484 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:23:45.825484 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 29 21:23:45.825484 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 29 21:23:45.825484 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 29 21:23:45.825484 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 29 21:23:45.826607 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:23:45.826607 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:23:45.826607 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 29 21:23:45.826607 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 21:23:45.826607 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 29 21:23:45.826607 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 21:23:45.826607 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:23:45.826607 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:23:45.826607 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:23:45.826607 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:23:45.826607 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:23:45.826607 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:23:45.826607 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:23:45.826607 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:23:45.826607 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:23:45.826607 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:23:45.826607 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:23:45.826607 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:23:45.827591 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:23:45.827591 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:23:45.827591 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 29 21:23:45.827591 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 29 21:23:45.827591 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 29 21:23:45.827591 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 29 21:23:45.827591 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 29 21:23:45.827591 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 29 21:23:45.827591 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Aug 29 21:23:45.827591 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:23:45.827591 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 614/1226] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:23:45 2022] GET /identity/v3/users/placement => generated 87 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:23:45.833149 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d871fc0a-9c27-458d-ad52-c301cd22039e None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:23:45.896382 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:23:45.955680 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d871fc0a-9c27-458d-ad52-c301cd22039e None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:23:45.992286 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec32f942-7287-4a9e-9858-b28427586e4d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:23:45.993021 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec32f942-7287-4a9e-9858-b28427586e4d None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:23:45.993406 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec32f942-7287-4a9e-9858-b28427586e4d None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:23:45.993757 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec32f942-7287-4a9e-9858-b28427586e4d None admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:23:45.994613 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec32f942-7287-4a9e-9858-b28427586e4d None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:23:45.995022 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec32f942-7287-4a9e-9858-b28427586e4d None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:23:45.997200 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec32f942-7287-4a9e-9858-b28427586e4d None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:23:46.017886 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 613/1227] 10.222.0.90 () {56 vars in 1143 bytes} [Mon Aug 29 21:23:45 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:23:46.021471 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a49ad93-dd22-4c97-8b5a-597d8c81b15c None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:23:46.132667 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a49ad93-dd22-4c97-8b5a-597d8c81b15c None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:23:46.167075 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b19a00d4-4242-4e1d-a444-f0edd0eb689b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:23:46.168028 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b19a00d4-4242-4e1d-a444-f0edd0eb689b None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:23:46.168028 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b19a00d4-4242-4e1d-a444-f0edd0eb689b None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:23:46.168400 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b19a00d4-4242-4e1d-a444-f0edd0eb689b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:23:46.179574 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b19a00d4-4242-4e1d-a444-f0edd0eb689b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:23:46.181058 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b19a00d4-4242-4e1d-a444-f0edd0eb689b None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:23:46.196000 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-b19a00d4-4242-4e1d-a444-f0edd0eb689b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:23:46.196000 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:23:46.196000 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:23:46.196000 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:23:46.196000 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:23:46.196000 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:23:46.196000 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:23:46.196000 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:23:46.196000 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:23:46.196000 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:23:46.196000 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:23:46.196000 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:23:46.196000 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 21:23:46.196000 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 21:23:46.196000 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 21:23:46.196000 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 21:23:46.197256 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:23:46.197256 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:23:46.197256 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:23:46.197256 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:23:46.197256 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:23:46.197256 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:23:46.197256 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:23:46.197256 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:23:46.197256 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:23:46.197256 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:23:46.197256 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:23:46.197256 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:23:46.197256 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:23:46.197256 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:23:46.197256 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:23:46.197256 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:23:46.197256 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 21:23:46.198302 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 21:23:46.198302 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:23:46.198302 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:23:46.198302 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 615/1228] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:23:46 2022] GET /identity/v3/domains/Default => generated 87 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:23:46.204403 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec32f942-7287-4a9e-9858-b28427586e4d None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:23:46.321530 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec32f942-7287-4a9e-9858-b28427586e4d None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:23:46.324808 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:23:46.361537 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bb1e80f-b066-4ad0-8d9b-2d237535a885 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:23:46.362262 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb1e80f-b066-4ad0-8d9b-2d237535a885 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:23:46.362564 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb1e80f-b066-4ad0-8d9b-2d237535a885 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:23:46.362900 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb1e80f-b066-4ad0-8d9b-2d237535a885 None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:23:46.363593 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bb1e80f-b066-4ad0-8d9b-2d237535a885 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:23:46.363919 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bb1e80f-b066-4ad0-8d9b-2d237535a885 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:23:46.365382 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bb1e80f-b066-4ad0-8d9b-2d237535a885 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:23:46.379144 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 614/1229] 10.222.0.90 () {56 vars in 1109 bytes} [Mon Aug 29 21:23:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:23:46.385225 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b19a00d4-4242-4e1d-a444-f0edd0eb689b None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:23:46.396044 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:23:46.498892 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b19a00d4-4242-4e1d-a444-f0edd0eb689b None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:23:46.535803 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93347030-8673-4c16-bdc1-ce51bbf26807 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:23:46.536447 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93347030-8673-4c16-bdc1-ce51bbf26807 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:23:46.536731 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93347030-8673-4c16-bdc1-ce51bbf26807 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:23:46.537006 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93347030-8673-4c16-bdc1-ce51bbf26807 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:23:46.544619 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93347030-8673-4c16-bdc1-ce51bbf26807 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:23:46.546095 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93347030-8673-4c16-bdc1-ce51bbf26807 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:23:46.557662 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-93347030-8673-4c16-bdc1-ce51bbf26807 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 21:23:46.557662 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:23:46.557662 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:23:46.557662 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:23:46.557662 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:23:46.557662 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:23:46.557662 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:23:46.557662 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:23:46.557662 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:23:46.557662 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:23:46.557662 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:23:46.557662 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:23:46.557662 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 29 21:23:46.557662 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 29 21:23:46.557662 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 29 21:23:46.565269 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 29 21:23:46.565269 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:23:46.565269 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:23:46.565269 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:23:46.565269 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:23:46.565269 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:23:46.565269 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:23:46.565269 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:23:46.565269 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:23:46.565269 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:23:46.565269 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:23:46.565269 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:23:46.565269 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:23:46.565269 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:23:46.565269 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:23:46.565269 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:23:46.565269 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:23:46.566506 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 29 21:23:46.566506 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 29 21:23:46.566506 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 29 21:23:46.566506 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 29 21:23:46.566506 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 29 21:23:46.566506 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 29 21:23:46.566506 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 21:23:46.566506 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:23:46.566506 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 616/1230] 10.222.0.90 () {56 vars in 1111 bytes} [Mon Aug 29 21:23:46 2022] GET /identity/v3/projects/service => generated 88 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 29 21:23:46.568850 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bb1e80f-b066-4ad0-8d9b-2d237535a885 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:23:46.677982 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bb1e80f-b066-4ad0-8d9b-2d237535a885 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:23:46.712682 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90218d3b-3968-4c7c-a69b-915431d1eb86 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:23:46.713302 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90218d3b-3968-4c7c-a69b-915431d1eb86 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:23:46.713582 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90218d3b-3968-4c7c-a69b-915431d1eb86 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:23:46.713883 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90218d3b-3968-4c7c-a69b-915431d1eb86 None admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:23:46.715172 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90218d3b-3968-4c7c-a69b-915431d1eb86 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:23:46.715537 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90218d3b-3968-4c7c-a69b-915431d1eb86 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:23:46.716928 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90218d3b-3968-4c7c-a69b-915431d1eb86 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:23:46.728977 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 615/1231] 10.222.0.90 () {56 vars in 1148 bytes} [Mon Aug 29 21:23:46 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:23:46.736241 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93347030-8673-4c16-bdc1-ce51bbf26807 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:23:46.816416 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:23:46.857577 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93347030-8673-4c16-bdc1-ce51bbf26807 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:23:46.896697 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab10bfb2-eb2a-4b0c-949c-8fa428ca916b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:23:46.897576 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab10bfb2-eb2a-4b0c-949c-8fa428ca916b None admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:23:46.897916 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab10bfb2-eb2a-4b0c-949c-8fa428ca916b None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:23:46.898194 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab10bfb2-eb2a-4b0c-949c-8fa428ca916b None admin] PATH_INFO: `/v3/projects/ec03e112e3d547fbab6faf7b485c3fca/users/b95fa07a7c0549dbaac49406e2639e21/roles/a94bdb4c5796428baf63644c062dfcb2` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:23:46.916767 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab10bfb2-eb2a-4b0c-949c-8fa428ca916b None admin] RBAC: Authorizing `identity:create_grant(project_id=ec03e112e3d547fbab6faf7b485c3fca, user_id=b95fa07a7c0549dbaac49406e2639e21, role_id=a94bdb4c5796428baf63644c062dfcb2)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:23:46.919049 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab10bfb2-eb2a-4b0c-949c-8fa428ca916b None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:23:46.934136 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 617/1232] 10.222.0.90 () {58 vars in 1439 bytes} [Mon Aug 29 21:23:46 2022] PUT /identity/v3/projects/ec03e112e3d547fbab6faf7b485c3fca/users/b95fa07a7c0549dbaac49406e2639e21/roles/a94bdb4c5796428baf63644c062dfcb2 => generated 0 bytes in 198 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 21:23:49.109203 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b82eecfe-1702-4a18-9501-34c088fc8f30 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:23:49.109734 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b82eecfe-1702-4a18-9501-34c088fc8f30 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:23:49.109734 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b82eecfe-1702-4a18-9501-34c088fc8f30 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:23:49.111179 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 616/1233] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:23:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:23:49.117432 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bfc3dcb-11e5-49c4-9d20-749181b045cb None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:23:49.117875 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bfc3dcb-11e5-49c4-9d20-749181b045cb None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:23:49.118018 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bfc3dcb-11e5-49c4-9d20-749181b045cb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:23:49.137098 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-3bfc3dcb-11e5-49c4-9d20-749181b045cb None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:23:49.138316 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-3bfc3dcb-11e5-49c4-9d20-749181b045cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:23:49.209097 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 618/1234] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:23:49 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 21:23:49.509179 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b82eecfe-1702-4a18-9501-34c088fc8f30 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:23:49.536980 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:23:49.624887 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b82eecfe-1702-4a18-9501-34c088fc8f30 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:23:49.662654 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07670a01-1b34-4de3-848f-5d771aa41d82 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:23:49.663357 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07670a01-1b34-4de3-848f-5d771aa41d82 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:23:49.663686 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07670a01-1b34-4de3-848f-5d771aa41d82 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:23:49.663956 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07670a01-1b34-4de3-848f-5d771aa41d82 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:23:49.675328 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07670a01-1b34-4de3-848f-5d771aa41d82 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 21:23:49.680415 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-07670a01-1b34-4de3-848f-5d771aa41d82 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 29 21:23:49.680415 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:23:49.680415 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:23:49.680415 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:23:49.680415 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:23:49.680415 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:23:49.680415 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:23:49.680415 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:23:49.680415 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:23:49.680415 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:23:49.680415 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:23:49.680415 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:23:49.680415 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 21:23:49.680415 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 21:23:49.680415 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 21:23:49.680415 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 21:23:49.681641 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 21:23:49.681641 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 21:23:49.681641 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 21:23:49.681641 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 21:23:49.681641 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:23:49.681641 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:23:49.681641 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:23:49.681641 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:23:49.681641 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:23:49.681641 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:23:49.681641 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:23:49.681641 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:23:49.681641 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:23:49.681641 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:23:49.681641 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:23:49.681641 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:23:49.682563 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:23:49.682563 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:23:49.682563 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:23:49.682563 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:23:49.682563 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 21:23:49.682563 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 21:23:49.682563 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 21:23:49.682563 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 21:23:49.682563 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 21:23:49.682563 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 21:23:49.682563 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 29 21:23:49.682563 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:23:49.682563 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 617/1235] 10.222.0.90 () {56 vars in 1102 bytes} [Mon Aug 29 21:23:49 2022] GET /identity/v3/roles/service => generated 85 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:23:49.688507 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bfc3dcb-11e5-49c4-9d20-749181b045cb None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:23:49.738321 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:23:49.802241 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bfc3dcb-11e5-49c4-9d20-749181b045cb None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:23:49.844774 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0edf6b1-9d31-4dab-90e8-a1b08d1699eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:23:49.845530 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0edf6b1-9d31-4dab-90e8-a1b08d1699eb None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:23:49.845931 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0edf6b1-9d31-4dab-90e8-a1b08d1699eb None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:23:49.846163 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0edf6b1-9d31-4dab-90e8-a1b08d1699eb None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:23:49.846961 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0edf6b1-9d31-4dab-90e8-a1b08d1699eb None admin] RBAC: Adding query filter params (name=service) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:23:49.847300 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0edf6b1-9d31-4dab-90e8-a1b08d1699eb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:23:49.848857 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0edf6b1-9d31-4dab-90e8-a1b08d1699eb None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:23:49.858434 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 619/1236] 10.222.0.90 () {56 vars in 1103 bytes} [Mon Aug 29 21:23:49 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:23:49.865160 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07670a01-1b34-4de3-848f-5d771aa41d82 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:23:49.966883 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:23:49.984362 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07670a01-1b34-4de3-848f-5d771aa41d82 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:23:50.027565 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bbaf004-9d9f-46aa-b2df-fd3ec6808dac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:23:50.028250 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bbaf004-9d9f-46aa-b2df-fd3ec6808dac None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:23:50.028605 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bbaf004-9d9f-46aa-b2df-fd3ec6808dac None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:23:50.028946 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bbaf004-9d9f-46aa-b2df-fd3ec6808dac None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:23:50.042447 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bbaf004-9d9f-46aa-b2df-fd3ec6808dac None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:23:50.044364 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bbaf004-9d9f-46aa-b2df-fd3ec6808dac None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:23:50.065956 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-8bbaf004-9d9f-46aa-b2df-fd3ec6808dac None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:23:50.065956 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:23:50.065956 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:23:50.065956 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:23:50.065956 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:23:50.065956 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:23:50.065956 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:23:50.065956 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:23:50.065956 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:23:50.065956 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:23:50.065956 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:23:50.065956 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:23:50.065956 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 21:23:50.065956 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 21:23:50.065956 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 21:23:50.065956 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 21:23:50.067970 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:23:50.067970 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:23:50.067970 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:23:50.067970 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:23:50.067970 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:23:50.067970 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:23:50.067970 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:23:50.067970 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:23:50.067970 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:23:50.067970 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:23:50.067970 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:23:50.067970 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:23:50.067970 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:23:50.067970 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:23:50.067970 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:23:50.067970 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:23:50.067970 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 21:23:50.069708 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 21:23:50.069708 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:23:50.069708 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:23:50.075907 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 618/1237] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:23:49 2022] GET /identity/v3/domains/Default => generated 87 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:23:50.076862 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0edf6b1-9d31-4dab-90e8-a1b08d1699eb None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:23:50.186376 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0edf6b1-9d31-4dab-90e8-a1b08d1699eb None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:23:50.199988 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:23:50.232188 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cc850ca-fabc-4c94-bf2a-11f9d34d45a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:23:50.232775 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cc850ca-fabc-4c94-bf2a-11f9d34d45a7 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:23:50.233050 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cc850ca-fabc-4c94-bf2a-11f9d34d45a7 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:23:50.233338 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cc850ca-fabc-4c94-bf2a-11f9d34d45a7 None admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:23:50.234272 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cc850ca-fabc-4c94-bf2a-11f9d34d45a7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:23:50.234612 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cc850ca-fabc-4c94-bf2a-11f9d34d45a7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:23:50.236212 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cc850ca-fabc-4c94-bf2a-11f9d34d45a7 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:23:50.250560 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 620/1238] 10.222.0.90 () {56 vars in 1109 bytes} [Mon Aug 29 21:23:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:23:50.256317 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bbaf004-9d9f-46aa-b2df-fd3ec6808dac None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:23:50.367628 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bbaf004-9d9f-46aa-b2df-fd3ec6808dac None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:23:50.412360 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-339a2584-93d3-44a1-bc34-eb6a20e7b758 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:23:50.413656 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-339a2584-93d3-44a1-bc34-eb6a20e7b758 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:23:50.414096 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-339a2584-93d3-44a1-bc34-eb6a20e7b758 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:23:50.414690 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-339a2584-93d3-44a1-bc34-eb6a20e7b758 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:23:50.418707 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:23:50.426734 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-339a2584-93d3-44a1-bc34-eb6a20e7b758 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:23:50.429155 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-339a2584-93d3-44a1-bc34-eb6a20e7b758 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:23:50.448615 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-339a2584-93d3-44a1-bc34-eb6a20e7b758 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Aug 29 21:23:50.448615 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:23:50.448615 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:23:50.448615 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:23:50.448615 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:23:50.448615 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:23:50.448615 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:23:50.448615 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:23:50.448615 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:23:50.448615 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:23:50.448615 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:23:50.448615 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:23:50.448615 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 29 21:23:50.448615 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 29 21:23:50.448615 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 29 21:23:50.448615 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 29 21:23:50.449941 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:23:50.449941 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:23:50.449941 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 29 21:23:50.449941 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 21:23:50.449941 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 29 21:23:50.449941 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 21:23:50.449941 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:23:50.449941 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:23:50.449941 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:23:50.449941 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:23:50.449941 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:23:50.449941 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:23:50.449941 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:23:50.449941 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:23:50.449941 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:23:50.449941 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:23:50.449941 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:23:50.449941 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:23:50.451154 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:23:50.451154 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:23:50.451154 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 29 21:23:50.451154 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 29 21:23:50.451154 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 29 21:23:50.451154 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 29 21:23:50.451154 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 29 21:23:50.451154 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 29 21:23:50.451154 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Aug 29 21:23:50.451154 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:23:50.451154 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 619/1239] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:23:50 2022] GET /identity/v3/users/placement => generated 87 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:23:50.457337 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cc850ca-fabc-4c94-bf2a-11f9d34d45a7 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:23:50.599077 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cc850ca-fabc-4c94-bf2a-11f9d34d45a7 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:23:50.636111 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91ba7c41-4ac6-43dc-b548-750b2fac72ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:23:50.636642 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91ba7c41-4ac6-43dc-b548-750b2fac72ad None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:23:50.636966 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91ba7c41-4ac6-43dc-b548-750b2fac72ad None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:23:50.637297 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91ba7c41-4ac6-43dc-b548-750b2fac72ad None admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:23:50.638239 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91ba7c41-4ac6-43dc-b548-750b2fac72ad None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:23:50.638669 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91ba7c41-4ac6-43dc-b548-750b2fac72ad None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:23:50.640367 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91ba7c41-4ac6-43dc-b548-750b2fac72ad None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:23:50.661604 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 621/1240] 10.222.0.90 () {56 vars in 1143 bytes} [Mon Aug 29 21:23:50 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:23:50.668514 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-339a2584-93d3-44a1-bc34-eb6a20e7b758 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:23:50.786633 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-339a2584-93d3-44a1-bc34-eb6a20e7b758 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:23:50.822947 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baa43412-1f3c-48dc-a49e-f3de66c8faf9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:23:50.823734 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa43412-1f3c-48dc-a49e-f3de66c8faf9 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:23:50.824014 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa43412-1f3c-48dc-a49e-f3de66c8faf9 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:23:50.824287 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa43412-1f3c-48dc-a49e-f3de66c8faf9 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:23:50.888800 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:23:50.896561 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baa43412-1f3c-48dc-a49e-f3de66c8faf9 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:23:50.898454 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baa43412-1f3c-48dc-a49e-f3de66c8faf9 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:23:51.022875 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 620/1241] 10.222.0.90 () {58 vars in 1282 bytes} [Mon Aug 29 21:23:50 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 355 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 29 21:23:51.029728 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91ba7c41-4ac6-43dc-b548-750b2fac72ad None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:23:51.059601 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:23:51.143566 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91ba7c41-4ac6-43dc-b548-750b2fac72ad None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:23:51.181941 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-373c2e75-68d2-4324-9b7c-085548d88603 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:23:51.182807 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-373c2e75-68d2-4324-9b7c-085548d88603 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:23:51.183223 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-373c2e75-68d2-4324-9b7c-085548d88603 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:23:51.183570 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-373c2e75-68d2-4324-9b7c-085548d88603 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:23:51.196236 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-373c2e75-68d2-4324-9b7c-085548d88603 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:23:51.197692 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-373c2e75-68d2-4324-9b7c-085548d88603 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:23:51.215627 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-373c2e75-68d2-4324-9b7c-085548d88603 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:23:51.215627 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:23:51.215627 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:23:51.215627 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:23:51.215627 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:23:51.215627 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:23:51.215627 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:23:51.215627 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:23:51.215627 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:23:51.215627 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:23:51.215627 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:23:51.215627 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:23:51.215627 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 21:23:51.215627 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 21:23:51.215627 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 21:23:51.215627 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 21:23:51.216820 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:23:51.216820 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:23:51.216820 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:23:51.216820 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:23:51.216820 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:23:51.216820 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:23:51.216820 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:23:51.216820 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:23:51.216820 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:23:51.216820 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:23:51.216820 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:23:51.216820 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:23:51.216820 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:23:51.216820 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:23:51.216820 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:23:51.216820 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:23:51.216820 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 21:23:51.218525 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 21:23:51.218525 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:23:51.218525 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:23:51.223252 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 622/1242] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:23:51 2022] GET /identity/v3/domains/Default => generated 87 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 29 21:23:51.224470 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baa43412-1f3c-48dc-a49e-f3de66c8faf9 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:23:51.333097 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:23:51.340356 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baa43412-1f3c-48dc-a49e-f3de66c8faf9 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:23:51.379310 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67dddbdc-a3ae-4d86-80df-f2ddf92ebe3e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:23:51.380026 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67dddbdc-a3ae-4d86-80df-f2ddf92ebe3e None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:23:51.380342 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67dddbdc-a3ae-4d86-80df-f2ddf92ebe3e None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:23:51.380625 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67dddbdc-a3ae-4d86-80df-f2ddf92ebe3e None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:23:51.381390 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67dddbdc-a3ae-4d86-80df-f2ddf92ebe3e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:23:51.381763 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67dddbdc-a3ae-4d86-80df-f2ddf92ebe3e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:23:51.383314 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67dddbdc-a3ae-4d86-80df-f2ddf92ebe3e None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:23:51.397458 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 621/1243] 10.222.0.90 () {56 vars in 1109 bytes} [Mon Aug 29 21:23:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:23:51.403813 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-373c2e75-68d2-4324-9b7c-085548d88603 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:23:51.497403 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:23:51.523717 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-373c2e75-68d2-4324-9b7c-085548d88603 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:23:51.561288 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-800a13e1-d33c-4515-917f-a51317dfc54f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:23:51.561950 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-800a13e1-d33c-4515-917f-a51317dfc54f None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:23:51.562233 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-800a13e1-d33c-4515-917f-a51317dfc54f None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:23:51.562510 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-800a13e1-d33c-4515-917f-a51317dfc54f None admin] PATH_INFO: `/v3/projects/service` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:23:51.569585 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-800a13e1-d33c-4515-917f-a51317dfc54f None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:23:51.571132 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-800a13e1-d33c-4515-917f-a51317dfc54f None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:23:51.586503 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-800a13e1-d33c-4515-917f-a51317dfc54f None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 21:23:51.586503 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:23:51.586503 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:23:51.586503 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:23:51.586503 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:23:51.586503 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:23:51.586503 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:23:51.586503 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:23:51.586503 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:23:51.586503 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:23:51.586503 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:23:51.586503 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:23:51.586503 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 29 21:23:51.586503 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 29 21:23:51.586503 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 29 21:23:51.588646 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 29 21:23:51.588646 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:23:51.588646 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:23:51.588646 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:23:51.588646 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:23:51.588646 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:23:51.588646 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:23:51.588646 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:23:51.588646 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:23:51.588646 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:23:51.588646 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:23:51.588646 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:23:51.588646 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:23:51.588646 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:23:51.588646 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:23:51.588646 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:23:51.588646 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:23:51.591332 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 29 21:23:51.591332 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 29 21:23:51.591332 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 29 21:23:51.591332 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 29 21:23:51.591332 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 29 21:23:51.591332 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 29 21:23:51.591332 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 21:23:51.591332 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:23:51.591332 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 623/1244] 10.222.0.90 () {56 vars in 1111 bytes} [Mon Aug 29 21:23:51 2022] GET /identity/v3/projects/service => generated 88 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:23:51.596853 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67dddbdc-a3ae-4d86-80df-f2ddf92ebe3e None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:23:51.706964 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67dddbdc-a3ae-4d86-80df-f2ddf92ebe3e None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:23:51.745296 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92831eef-9227-49fc-9bc5-3e341ad8edf7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:23:51.746158 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92831eef-9227-49fc-9bc5-3e341ad8edf7 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:23:51.746489 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92831eef-9227-49fc-9bc5-3e341ad8edf7 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:23:51.746764 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92831eef-9227-49fc-9bc5-3e341ad8edf7 None admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:23:51.747647 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92831eef-9227-49fc-9bc5-3e341ad8edf7 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:23:51.747951 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92831eef-9227-49fc-9bc5-3e341ad8edf7 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:23:51.749528 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92831eef-9227-49fc-9bc5-3e341ad8edf7 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:23:51.764412 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 622/1245] 10.222.0.90 () {56 vars in 1148 bytes} [Mon Aug 29 21:23:51 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:23:51.771493 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-800a13e1-d33c-4515-917f-a51317dfc54f None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:23:51.909264 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-800a13e1-d33c-4515-917f-a51317dfc54f None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:23:51.946797 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e97f1aeb-c33d-4dc2-8977-ddf31cadc8b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:23:51.947519 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e97f1aeb-c33d-4dc2-8977-ddf31cadc8b8 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:23:51.947807 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e97f1aeb-c33d-4dc2-8977-ddf31cadc8b8 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:23:51.948075 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e97f1aeb-c33d-4dc2-8977-ddf31cadc8b8 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:23:51.949395 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e97f1aeb-c33d-4dc2-8977-ddf31cadc8b8 None admin] RBAC: Adding query filter params (role.id=a94bdb4c5796428baf63644c062dfcb2, scope.project.id=ec03e112e3d547fbab6faf7b485c3fca, user.id=b95fa07a7c0549dbaac49406e2639e21) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:23:51.949710 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e97f1aeb-c33d-4dc2-8977-ddf31cadc8b8 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:23:51.951173 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e97f1aeb-c33d-4dc2-8977-ddf31cadc8b8 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:23:51.957376 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 624/1246] 10.222.0.90 () {56 vars in 1374 bytes} [Mon Aug 29 21:23:51 2022] GET /identity/v3/role_assignments?user.id=b95fa07a7c0549dbaac49406e2639e21&scope.project.id=ec03e112e3d547fbab6faf7b485c3fca&role.id=a94bdb4c5796428baf63644c062dfcb2 => generated 608 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:23:54.261756 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c5eaaec-2d2c-463c-9445-7cb77b39fddd None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:23:54.262422 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c5eaaec-2d2c-463c-9445-7cb77b39fddd None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:23:54.262422 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c5eaaec-2d2c-463c-9445-7cb77b39fddd None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:23:54.263686 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 623/1247] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:23:54 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:23:54.270797 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3de6b6b-32ee-468f-88a8-406eaf0670c1 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:23:54.270797 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3de6b6b-32ee-468f-88a8-406eaf0670c1 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:23:54.271586 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3de6b6b-32ee-468f-88a8-406eaf0670c1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:23:54.279592 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:23:54.294492 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-d3de6b6b-32ee-468f-88a8-406eaf0670c1 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:23:54.295930 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-d3de6b6b-32ee-468f-88a8-406eaf0670c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:23:54.373846 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 625/1248] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:23:54 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 21:23:54.696355 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c5eaaec-2d2c-463c-9445-7cb77b39fddd None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:23:54.715466 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:23:54.814156 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c5eaaec-2d2c-463c-9445-7cb77b39fddd None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:23:54.849159 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57fe2b4f-1cd1-4f92-855d-4b3a420c6b4e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:23:54.849934 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57fe2b4f-1cd1-4f92-855d-4b3a420c6b4e None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:23:54.850187 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57fe2b4f-1cd1-4f92-855d-4b3a420c6b4e None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:23:54.850397 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57fe2b4f-1cd1-4f92-855d-4b3a420c6b4e None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:23:54.866865 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57fe2b4f-1cd1-4f92-855d-4b3a420c6b4e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 21:23:54.872429 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-57fe2b4f-1cd1-4f92-855d-4b3a420c6b4e None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 21:23:54.872429 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:23:54.872429 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:23:54.872429 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:23:54.872429 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:23:54.872429 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:23:54.872429 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:23:54.872429 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:23:54.872429 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:23:54.872429 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:23:54.872429 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:23:54.872429 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:23:54.872429 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 21:23:54.872429 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 21:23:54.872429 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 21:23:54.872429 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 21:23:54.873476 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 21:23:54.873476 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 21:23:54.873476 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 21:23:54.873476 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 21:23:54.873476 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:23:54.873476 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:23:54.873476 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:23:54.873476 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:23:54.873476 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:23:54.873476 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:23:54.873476 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:23:54.873476 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:23:54.873476 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:23:54.873476 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:23:54.873476 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:23:54.873476 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:23:54.874545 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:23:54.874545 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:23:54.874545 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:23:54.874545 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:23:54.874545 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 21:23:54.874545 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 21:23:54.874545 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 21:23:54.874545 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 21:23:54.874545 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 21:23:54.874545 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 21:23:54.874545 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 21:23:54.874545 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:23:54.874545 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 624/1249] 10.222.0.90 () {56 vars in 1096 bytes} [Mon Aug 29 21:23:54 2022] GET /identity/v3/roles/admin => generated 83 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:23:54.881570 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3de6b6b-32ee-468f-88a8-406eaf0670c1 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:23:55.002541 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3de6b6b-32ee-468f-88a8-406eaf0670c1 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:23:55.038078 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:23:55.045138 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-174ba94a-d2e4-4941-b7e6-c54a5c6eb6b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:23:55.045819 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-174ba94a-d2e4-4941-b7e6-c54a5c6eb6b6 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:23:55.046255 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-174ba94a-d2e4-4941-b7e6-c54a5c6eb6b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:23:55.046533 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-174ba94a-d2e4-4941-b7e6-c54a5c6eb6b6 None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:23:55.047210 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-174ba94a-d2e4-4941-b7e6-c54a5c6eb6b6 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:23:55.047521 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-174ba94a-d2e4-4941-b7e6-c54a5c6eb6b6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:23:55.049076 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-174ba94a-d2e4-4941-b7e6-c54a5c6eb6b6 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:23:55.059834 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 626/1250] 10.222.0.90 () {56 vars in 1099 bytes} [Mon Aug 29 21:23:54 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:23:55.066055 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57fe2b4f-1cd1-4f92-855d-4b3a420c6b4e None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:23:55.166998 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:23:55.197510 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57fe2b4f-1cd1-4f92-855d-4b3a420c6b4e None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:23:55.247028 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f2415f9-fe4a-4161-8874-35c89348085e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:23:55.247696 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f2415f9-fe4a-4161-8874-35c89348085e None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:23:55.247980 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f2415f9-fe4a-4161-8874-35c89348085e None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:23:55.248261 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f2415f9-fe4a-4161-8874-35c89348085e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:23:55.261970 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f2415f9-fe4a-4161-8874-35c89348085e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:23:55.263824 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f2415f9-fe4a-4161-8874-35c89348085e None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:23:55.283203 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-5f2415f9-fe4a-4161-8874-35c89348085e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:23:55.283203 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:23:55.283203 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:23:55.283203 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:23:55.283203 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:23:55.283203 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:23:55.283203 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:23:55.283203 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:23:55.283203 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:23:55.283203 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:23:55.283203 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:23:55.283203 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:23:55.283203 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 21:23:55.283203 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 21:23:55.283203 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 21:23:55.283203 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 21:23:55.284444 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:23:55.284444 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:23:55.284444 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:23:55.284444 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:23:55.284444 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:23:55.284444 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:23:55.284444 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:23:55.284444 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:23:55.284444 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:23:55.284444 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:23:55.284444 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:23:55.284444 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:23:55.284444 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:23:55.284444 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:23:55.284444 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:23:55.284444 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:23:55.284444 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 21:23:55.285652 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 21:23:55.285652 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:23:55.285652 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:23:55.285652 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 625/1251] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:23:55 2022] GET /identity/v3/domains/Default => generated 87 bytes in 219 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:23:55.291291 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-174ba94a-d2e4-4941-b7e6-c54a5c6eb6b6 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:23:55.402385 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-174ba94a-d2e4-4941-b7e6-c54a5c6eb6b6 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:23:55.444757 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a463b58-3547-4cb7-99f0-fa3b38c859e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:23:55.445488 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a463b58-3547-4cb7-99f0-fa3b38c859e7 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:23:55.445859 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a463b58-3547-4cb7-99f0-fa3b38c859e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:23:55.446207 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a463b58-3547-4cb7-99f0-fa3b38c859e7 None admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:23:55.446965 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a463b58-3547-4cb7-99f0-fa3b38c859e7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:23:55.447356 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a463b58-3547-4cb7-99f0-fa3b38c859e7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:23:55.449670 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a463b58-3547-4cb7-99f0-fa3b38c859e7 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:23:55.462453 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 627/1252] 10.222.0.90 () {56 vars in 1109 bytes} [Mon Aug 29 21:23:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:23:55.470039 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f2415f9-fe4a-4161-8874-35c89348085e None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:23:55.627418 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f2415f9-fe4a-4161-8874-35c89348085e None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:23:55.671540 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:23:55.684118 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10266469-56a7-43e2-8aa4-a366b2b77380 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:23:55.685190 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10266469-56a7-43e2-8aa4-a366b2b77380 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:23:55.685942 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10266469-56a7-43e2-8aa4-a366b2b77380 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:23:55.686614 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10266469-56a7-43e2-8aa4-a366b2b77380 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:23:55.700399 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10266469-56a7-43e2-8aa4-a366b2b77380 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:23:55.702449 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10266469-56a7-43e2-8aa4-a366b2b77380 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:23:55.721380 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-10266469-56a7-43e2-8aa4-a366b2b77380 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Aug 29 21:23:55.721380 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:23:55.721380 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:23:55.721380 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:23:55.721380 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:23:55.721380 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:23:55.721380 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:23:55.721380 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:23:55.721380 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:23:55.721380 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:23:55.721380 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:23:55.721380 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:23:55.721380 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 29 21:23:55.721380 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 29 21:23:55.721380 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 29 21:23:55.721380 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 29 21:23:55.722457 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:23:55.722457 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:23:55.722457 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 29 21:23:55.722457 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 21:23:55.722457 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 29 21:23:55.722457 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 21:23:55.722457 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:23:55.722457 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:23:55.722457 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:23:55.722457 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:23:55.722457 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:23:55.722457 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:23:55.722457 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:23:55.722457 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:23:55.722457 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:23:55.722457 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:23:55.722457 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:23:55.722457 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:23:55.723966 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:23:55.723966 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:23:55.723966 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 29 21:23:55.723966 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 29 21:23:55.723966 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 29 21:23:55.723966 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 29 21:23:55.723966 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 29 21:23:55.723966 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 29 21:23:55.723966 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Aug 29 21:23:55.723966 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:23:55.723966 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 626/1253] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:23:55 2022] GET /identity/v3/users/placement => generated 87 bytes in 253 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:23:55.729401 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a463b58-3547-4cb7-99f0-fa3b38c859e7 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:23:55.797425 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:23:55.849959 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a463b58-3547-4cb7-99f0-fa3b38c859e7 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:23:55.887370 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80b02f5d-4b15-49fc-b952-0f26d4e9fbc7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:23:55.887905 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80b02f5d-4b15-49fc-b952-0f26d4e9fbc7 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:23:55.888206 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80b02f5d-4b15-49fc-b952-0f26d4e9fbc7 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:23:55.888477 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80b02f5d-4b15-49fc-b952-0f26d4e9fbc7 None admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:23:55.889203 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80b02f5d-4b15-49fc-b952-0f26d4e9fbc7 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:23:55.889510 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80b02f5d-4b15-49fc-b952-0f26d4e9fbc7 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:23:55.891027 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80b02f5d-4b15-49fc-b952-0f26d4e9fbc7 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:23:55.910518 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 628/1254] 10.222.0.90 () {56 vars in 1143 bytes} [Mon Aug 29 21:23:55 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:23:55.916645 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10266469-56a7-43e2-8aa4-a366b2b77380 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:23:56.031444 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10266469-56a7-43e2-8aa4-a366b2b77380 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:23:56.067051 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fee7067-a7fd-4680-8ab6-12c4621b5eee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:23:56.067672 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fee7067-a7fd-4680-8ab6-12c4621b5eee None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:23:56.067951 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fee7067-a7fd-4680-8ab6-12c4621b5eee None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:23:56.068225 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fee7067-a7fd-4680-8ab6-12c4621b5eee None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:23:56.136953 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:23:56.154530 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fee7067-a7fd-4680-8ab6-12c4621b5eee None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:23:56.155905 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fee7067-a7fd-4680-8ab6-12c4621b5eee None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:23:56.268894 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 627/1255] 10.222.0.90 () {58 vars in 1282 bytes} [Mon Aug 29 21:23:55 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 353 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 29 21:23:56.275120 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80b02f5d-4b15-49fc-b952-0f26d4e9fbc7 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:23:56.390060 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80b02f5d-4b15-49fc-b952-0f26d4e9fbc7 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:23:56.406421 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:23:56.432815 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62492e02-f27d-4b8f-8b4c-ad89fb931d9c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:23:56.433743 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62492e02-f27d-4b8f-8b4c-ad89fb931d9c None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:23:56.434064 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62492e02-f27d-4b8f-8b4c-ad89fb931d9c None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:23:56.434341 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62492e02-f27d-4b8f-8b4c-ad89fb931d9c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:23:56.445817 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62492e02-f27d-4b8f-8b4c-ad89fb931d9c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:23:56.447310 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62492e02-f27d-4b8f-8b4c-ad89fb931d9c None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:23:56.468694 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-62492e02-f27d-4b8f-8b4c-ad89fb931d9c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:23:56.468694 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:23:56.468694 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:23:56.468694 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:23:56.468694 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:23:56.468694 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:23:56.468694 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:23:56.468694 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:23:56.468694 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:23:56.468694 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:23:56.468694 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:23:56.468694 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:23:56.468694 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 21:23:56.468694 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 21:23:56.468694 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 21:23:56.468694 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 21:23:56.470204 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:23:56.470204 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:23:56.470204 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:23:56.470204 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:23:56.470204 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:23:56.470204 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:23:56.470204 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:23:56.470204 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:23:56.470204 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:23:56.470204 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:23:56.470204 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:23:56.470204 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:23:56.470204 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:23:56.470204 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:23:56.470204 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:23:56.470204 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:23:56.470204 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 21:23:56.471437 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 21:23:56.471437 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:23:56.471437 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:23:56.471437 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 629/1256] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:23:56 2022] GET /identity/v3/domains/Default => generated 87 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:23:56.476551 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fee7067-a7fd-4680-8ab6-12c4621b5eee None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:23:56.584146 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:23:56.599493 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fee7067-a7fd-4680-8ab6-12c4621b5eee None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:23:56.637592 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdb771b7-6d16-439b-8215-099550db5d9f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:23:56.638301 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdb771b7-6d16-439b-8215-099550db5d9f None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:23:56.638581 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdb771b7-6d16-439b-8215-099550db5d9f None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:23:56.638855 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdb771b7-6d16-439b-8215-099550db5d9f None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:23:56.639509 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdb771b7-6d16-439b-8215-099550db5d9f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:23:56.639813 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdb771b7-6d16-439b-8215-099550db5d9f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:23:56.641187 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdb771b7-6d16-439b-8215-099550db5d9f None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:23:56.654683 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 628/1257] 10.222.0.90 () {56 vars in 1109 bytes} [Mon Aug 29 21:23:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:23:56.660487 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62492e02-f27d-4b8f-8b4c-ad89fb931d9c None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:23:56.770424 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62492e02-f27d-4b8f-8b4c-ad89fb931d9c None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:23:56.806550 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af737b81-72f7-4438-8ca2-5259716d9bf5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:23:56.807197 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af737b81-72f7-4438-8ca2-5259716d9bf5 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:23:56.807520 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af737b81-72f7-4438-8ca2-5259716d9bf5 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:23:56.808240 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af737b81-72f7-4438-8ca2-5259716d9bf5 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:23:56.815193 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af737b81-72f7-4438-8ca2-5259716d9bf5 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:23:56.816644 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af737b81-72f7-4438-8ca2-5259716d9bf5 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:23:56.829083 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-af737b81-72f7-4438-8ca2-5259716d9bf5 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 21:23:56.829083 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:23:56.829083 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:23:56.829083 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:23:56.829083 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:23:56.829083 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:23:56.829083 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:23:56.829083 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:23:56.829083 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:23:56.829083 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:23:56.829083 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:23:56.829083 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:23:56.829083 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 29 21:23:56.829083 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 29 21:23:56.829083 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 29 21:23:56.830085 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 29 21:23:56.830085 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:23:56.830085 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:23:56.830085 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:23:56.830085 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:23:56.830085 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:23:56.830085 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:23:56.830085 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:23:56.830085 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:23:56.830085 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:23:56.830085 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:23:56.830085 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:23:56.830085 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:23:56.830085 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:23:56.830085 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:23:56.830085 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:23:56.830085 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:23:56.831028 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 29 21:23:56.831028 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 29 21:23:56.831028 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 29 21:23:56.831028 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 29 21:23:56.831028 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 29 21:23:56.831028 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 29 21:23:56.831028 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 21:23:56.831028 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:23:56.831028 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 630/1258] 10.222.0.90 () {56 vars in 1111 bytes} [Mon Aug 29 21:23:56 2022] GET /identity/v3/projects/service => generated 88 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 29 21:23:56.836087 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdb771b7-6d16-439b-8215-099550db5d9f None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:23:56.943769 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdb771b7-6d16-439b-8215-099550db5d9f None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:23:56.979893 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fa78470-4b37-49b4-b24b-a000d999d03c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:23:56.980610 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fa78470-4b37-49b4-b24b-a000d999d03c None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:23:56.980876 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fa78470-4b37-49b4-b24b-a000d999d03c None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:23:56.981149 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fa78470-4b37-49b4-b24b-a000d999d03c None admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:23:56.981899 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fa78470-4b37-49b4-b24b-a000d999d03c None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:23:56.982207 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fa78470-4b37-49b4-b24b-a000d999d03c None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:23:56.983634 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fa78470-4b37-49b4-b24b-a000d999d03c None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:23:56.997685 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 629/1259] 10.222.0.90 () {56 vars in 1148 bytes} [Mon Aug 29 21:23:56 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:23:57.003751 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af737b81-72f7-4438-8ca2-5259716d9bf5 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:23:57.031177 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:23:57.107463 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af737b81-72f7-4438-8ca2-5259716d9bf5 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:23:57.140459 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-507c961f-3737-4407-9aea-1157b4353e5d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:23:57.141061 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-507c961f-3737-4407-9aea-1157b4353e5d None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:23:57.141344 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-507c961f-3737-4407-9aea-1157b4353e5d None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:23:57.141619 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-507c961f-3737-4407-9aea-1157b4353e5d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:23:57.142369 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-507c961f-3737-4407-9aea-1157b4353e5d None admin] RBAC: Adding query filter params (role.id=a6e562cdc96c4be691d638990f8403fd, scope.project.id=ec03e112e3d547fbab6faf7b485c3fca, user.id=b95fa07a7c0549dbaac49406e2639e21) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:23:57.142678 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-507c961f-3737-4407-9aea-1157b4353e5d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:23:57.144101 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-507c961f-3737-4407-9aea-1157b4353e5d None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:23:57.149331 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 631/1260] 10.222.0.90 () {56 vars in 1374 bytes} [Mon Aug 29 21:23:57 2022] GET /identity/v3/role_assignments?user.id=b95fa07a7c0549dbaac49406e2639e21&scope.project.id=ec03e112e3d547fbab6faf7b485c3fca&role.id=a6e562cdc96c4be691d638990f8403fd => generated 259 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:23:59.025541 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9250955a-b445-4c3c-8c00-2829bd7facc0 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:23:59.026737 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9250955a-b445-4c3c-8c00-2829bd7facc0 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:23:59.026842 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9250955a-b445-4c3c-8c00-2829bd7facc0 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:23:59.028170 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 630/1261] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:23:59 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:23:59.033304 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1452a39c-4edc-462f-a36a-4e1ee4e93548 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:23:59.033702 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1452a39c-4edc-462f-a36a-4e1ee4e93548 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:23:59.033886 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1452a39c-4edc-462f-a36a-4e1ee4e93548 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:23:59.052556 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-1452a39c-4edc-462f-a36a-4e1ee4e93548 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:23:59.053747 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-1452a39c-4edc-462f-a36a-4e1ee4e93548 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:23:59.119101 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 632/1262] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:23:59 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 21:23:59.388967 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9250955a-b445-4c3c-8c00-2829bd7facc0 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:23:59.399872 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:23:59.498068 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9250955a-b445-4c3c-8c00-2829bd7facc0 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:23:59.538084 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5977442d-c46f-4fc1-bba8-a1560d82e439 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:23:59.538712 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5977442d-c46f-4fc1-bba8-a1560d82e439 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:23:59.538992 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5977442d-c46f-4fc1-bba8-a1560d82e439 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:23:59.539261 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5977442d-c46f-4fc1-bba8-a1560d82e439 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:23:59.549979 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5977442d-c46f-4fc1-bba8-a1560d82e439 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 21:23:59.554688 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-5977442d-c46f-4fc1-bba8-a1560d82e439 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 21:23:59.554688 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:23:59.554688 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:23:59.554688 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:23:59.554688 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:23:59.554688 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:23:59.554688 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:23:59.554688 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:23:59.554688 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:23:59.554688 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:23:59.554688 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:23:59.554688 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:23:59.554688 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 21:23:59.554688 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 21:23:59.554688 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 21:23:59.554688 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 21:23:59.555915 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 21:23:59.555915 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 21:23:59.555915 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 21:23:59.555915 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 21:23:59.555915 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:23:59.555915 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:23:59.555915 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:23:59.555915 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:23:59.555915 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:23:59.555915 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:23:59.555915 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:23:59.555915 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:23:59.555915 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:23:59.555915 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:23:59.555915 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:23:59.555915 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:23:59.557010 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:23:59.557010 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:23:59.557010 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:23:59.557010 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:23:59.557010 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 21:23:59.557010 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 21:23:59.557010 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 21:23:59.557010 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 21:23:59.557010 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 21:23:59.557010 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 21:23:59.557010 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 21:23:59.557010 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:23:59.557010 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 631/1263] 10.222.0.90 () {56 vars in 1096 bytes} [Mon Aug 29 21:23:59 2022] GET /identity/v3/roles/admin => generated 83 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:23:59.561820 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1452a39c-4edc-462f-a36a-4e1ee4e93548 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:23:59.579956 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:23:59.659224 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1452a39c-4edc-462f-a36a-4e1ee4e93548 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:23:59.692512 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f06e88a-d79e-456a-8d73-f154624a0f0b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:23:59.693114 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f06e88a-d79e-456a-8d73-f154624a0f0b None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:23:59.693396 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f06e88a-d79e-456a-8d73-f154624a0f0b None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:23:59.693671 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f06e88a-d79e-456a-8d73-f154624a0f0b None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:23:59.694360 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f06e88a-d79e-456a-8d73-f154624a0f0b None admin] RBAC: Adding query filter params (name=admin) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:23:59.694663 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f06e88a-d79e-456a-8d73-f154624a0f0b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:23:59.696021 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f06e88a-d79e-456a-8d73-f154624a0f0b None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:23:59.703857 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 633/1264] 10.222.0.90 () {56 vars in 1099 bytes} [Mon Aug 29 21:23:59 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:23:59.709583 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5977442d-c46f-4fc1-bba8-a1560d82e439 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:23:59.776604 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:23:59.809777 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5977442d-c46f-4fc1-bba8-a1560d82e439 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:23:59.842756 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dfc1bf1-d09b-4774-82b9-f084447a8b5a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:23:59.843382 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dfc1bf1-d09b-4774-82b9-f084447a8b5a None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:23:59.843659 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dfc1bf1-d09b-4774-82b9-f084447a8b5a None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:23:59.843929 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dfc1bf1-d09b-4774-82b9-f084447a8b5a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:23:59.855232 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dfc1bf1-d09b-4774-82b9-f084447a8b5a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:23:59.856675 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dfc1bf1-d09b-4774-82b9-f084447a8b5a None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:23:59.871704 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-2dfc1bf1-d09b-4774-82b9-f084447a8b5a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:23:59.871704 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:23:59.871704 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:23:59.871704 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:23:59.871704 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:23:59.871704 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:23:59.871704 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:23:59.871704 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:23:59.871704 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:23:59.871704 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:23:59.871704 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:23:59.871704 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:23:59.871704 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 21:23:59.871704 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 21:23:59.871704 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 21:23:59.871704 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 21:23:59.872883 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:23:59.872883 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:23:59.872883 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:23:59.872883 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:23:59.872883 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:23:59.872883 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:23:59.872883 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:23:59.872883 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:23:59.872883 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:23:59.872883 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:23:59.872883 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:23:59.872883 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:23:59.872883 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:23:59.872883 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:23:59.872883 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:23:59.872883 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:23:59.872883 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 21:23:59.874089 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 21:23:59.874089 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:23:59.874089 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:23:59.874089 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 632/1265] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:23:59 2022] GET /identity/v3/domains/Default => generated 87 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:23:59.878669 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f06e88a-d79e-456a-8d73-f154624a0f0b None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:23:59.957802 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:23:59.977514 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f06e88a-d79e-456a-8d73-f154624a0f0b None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:24:00.011599 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c50c5a2-77c2-4a72-9ee8-5dfc3e9b2e38 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:24:00.012195 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c50c5a2-77c2-4a72-9ee8-5dfc3e9b2e38 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:24:00.012476 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c50c5a2-77c2-4a72-9ee8-5dfc3e9b2e38 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:24:00.012748 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c50c5a2-77c2-4a72-9ee8-5dfc3e9b2e38 None admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:24:00.013419 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c50c5a2-77c2-4a72-9ee8-5dfc3e9b2e38 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:24:00.013681 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c50c5a2-77c2-4a72-9ee8-5dfc3e9b2e38 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:24:00.015070 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c50c5a2-77c2-4a72-9ee8-5dfc3e9b2e38 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:24:00.027135 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 634/1266] 10.222.0.90 () {56 vars in 1109 bytes} [Mon Aug 29 21:23:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:24:00.032953 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dfc1bf1-d09b-4774-82b9-f084447a8b5a None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:24:00.131502 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dfc1bf1-d09b-4774-82b9-f084447a8b5a None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:24:00.151889 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:24:00.165946 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68c655d8-9546-4920-87ac-6e716986d0bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:24:00.166665 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68c655d8-9546-4920-87ac-6e716986d0bd None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:24:00.167047 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68c655d8-9546-4920-87ac-6e716986d0bd None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:24:00.167399 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68c655d8-9546-4920-87ac-6e716986d0bd None admin] PATH_INFO: `/v3/users/placement` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:24:00.174660 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68c655d8-9546-4920-87ac-6e716986d0bd None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:24:00.176109 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68c655d8-9546-4920-87ac-6e716986d0bd None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:24:00.186890 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-68c655d8-9546-4920-87ac-6e716986d0bd None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Aug 29 21:24:00.186890 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:24:00.186890 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:24:00.186890 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:24:00.186890 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:24:00.186890 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:24:00.186890 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:24:00.186890 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:24:00.186890 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:24:00.186890 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:24:00.186890 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:24:00.186890 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:24:00.186890 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 29 21:24:00.186890 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 29 21:24:00.186890 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 29 21:24:00.186890 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 29 21:24:00.188046 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:24:00.188046 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:24:00.188046 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 29 21:24:00.188046 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 21:24:00.188046 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 29 21:24:00.188046 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 21:24:00.188046 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:24:00.188046 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:24:00.188046 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:24:00.188046 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:24:00.188046 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:24:00.188046 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:24:00.188046 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:24:00.188046 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:24:00.188046 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:24:00.188046 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:24:00.188046 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:24:00.188046 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:24:00.189459 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:24:00.189459 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:24:00.189459 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 29 21:24:00.189459 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 29 21:24:00.189459 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 29 21:24:00.189459 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 29 21:24:00.189459 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 29 21:24:00.189459 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 29 21:24:00.189459 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Aug 29 21:24:00.189459 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:24:00.189459 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 633/1267] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:24:00 2022] GET /identity/v3/users/placement => generated 87 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:24:00.193832 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c50c5a2-77c2-4a72-9ee8-5dfc3e9b2e38 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:24:00.293063 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c50c5a2-77c2-4a72-9ee8-5dfc3e9b2e38 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:24:00.326834 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2deab731-0d31-4100-bd03-e5bcdea1ba4c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:24:00.327442 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2deab731-0d31-4100-bd03-e5bcdea1ba4c None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:24:00.327861 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2deab731-0d31-4100-bd03-e5bcdea1ba4c None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:24:00.328170 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2deab731-0d31-4100-bd03-e5bcdea1ba4c None admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:24:00.328961 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2deab731-0d31-4100-bd03-e5bcdea1ba4c None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:24:00.329293 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2deab731-0d31-4100-bd03-e5bcdea1ba4c None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:24:00.330826 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2deab731-0d31-4100-bd03-e5bcdea1ba4c None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:24:00.339463 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:24:00.350150 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 635/1268] 10.222.0.90 () {56 vars in 1143 bytes} [Mon Aug 29 21:24:00 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:24:00.355881 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68c655d8-9546-4920-87ac-6e716986d0bd None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:24:00.454479 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68c655d8-9546-4920-87ac-6e716986d0bd None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:24:00.490989 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e75e0fcb-f171-4120-bdb3-6a67b99f9867 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:24:00.491695 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e75e0fcb-f171-4120-bdb3-6a67b99f9867 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:24:00.492405 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e75e0fcb-f171-4120-bdb3-6a67b99f9867 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:24:00.492405 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e75e0fcb-f171-4120-bdb3-6a67b99f9867 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:24:00.504752 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e75e0fcb-f171-4120-bdb3-6a67b99f9867 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:24:00.506419 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e75e0fcb-f171-4120-bdb3-6a67b99f9867 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:24:00.520945 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-e75e0fcb-f171-4120-bdb3-6a67b99f9867 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:24:00.520945 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:24:00.520945 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:24:00.520945 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:24:00.520945 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:24:00.520945 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:24:00.520945 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:24:00.520945 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:24:00.520945 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:24:00.520945 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:24:00.520945 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:24:00.520945 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:24:00.520945 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 21:24:00.520945 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 21:24:00.520945 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 21:24:00.520945 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 21:24:00.522148 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:24:00.522148 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:24:00.522148 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:24:00.522148 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:24:00.522148 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:24:00.522148 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:24:00.522148 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:24:00.522148 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:24:00.522148 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:24:00.522148 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:24:00.522148 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:24:00.522148 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:24:00.522148 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:24:00.522148 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:24:00.522148 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:24:00.522148 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:24:00.522148 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 21:24:00.523291 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 21:24:00.523291 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:24:00.523291 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:24:00.523291 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 634/1269] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:24:00 2022] GET /identity/v3/domains/Default => generated 87 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:24:00.528188 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2deab731-0d31-4100-bd03-e5bcdea1ba4c None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:24:00.628050 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2deab731-0d31-4100-bd03-e5bcdea1ba4c None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:24:00.661386 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9d2aaa3-2d10-4dfb-aeb5-99c5f99bf822 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:24:00.662166 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9d2aaa3-2d10-4dfb-aeb5-99c5f99bf822 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:24:00.662456 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9d2aaa3-2d10-4dfb-aeb5-99c5f99bf822 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:24:00.662732 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9d2aaa3-2d10-4dfb-aeb5-99c5f99bf822 None admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:24:00.663384 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9d2aaa3-2d10-4dfb-aeb5-99c5f99bf822 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:24:00.663695 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9d2aaa3-2d10-4dfb-aeb5-99c5f99bf822 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:24:00.665065 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9d2aaa3-2d10-4dfb-aeb5-99c5f99bf822 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:24:00.676906 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 636/1270] 10.222.0.90 () {56 vars in 1109 bytes} [Mon Aug 29 21:24:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:24:00.683285 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e75e0fcb-f171-4120-bdb3-6a67b99f9867 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:24:00.714601 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:24:00.782552 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e75e0fcb-f171-4120-bdb3-6a67b99f9867 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:24:00.816429 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31590252-2de4-4b8d-9095-054724a1fdb6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:24:00.817056 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31590252-2de4-4b8d-9095-054724a1fdb6 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:24:00.817336 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31590252-2de4-4b8d-9095-054724a1fdb6 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:24:00.817610 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31590252-2de4-4b8d-9095-054724a1fdb6 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:24:00.824215 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31590252-2de4-4b8d-9095-054724a1fdb6 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:24:00.825646 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31590252-2de4-4b8d-9095-054724a1fdb6 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:24:00.836660 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-31590252-2de4-4b8d-9095-054724a1fdb6 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 21:24:00.836660 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:24:00.836660 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:24:00.836660 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:24:00.836660 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:24:00.836660 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:24:00.836660 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:24:00.836660 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:24:00.836660 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:24:00.836660 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:24:00.836660 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:24:00.836660 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:24:00.836660 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 29 21:24:00.836660 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 29 21:24:00.836660 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 29 21:24:00.837731 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 29 21:24:00.837731 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:24:00.837731 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:24:00.837731 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:24:00.837731 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:24:00.837731 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:24:00.837731 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:24:00.837731 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:24:00.837731 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:24:00.837731 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:24:00.837731 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:24:00.837731 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:24:00.837731 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:24:00.837731 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:24:00.837731 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:24:00.837731 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:24:00.837731 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:24:00.838859 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 29 21:24:00.838859 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 29 21:24:00.838859 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 29 21:24:00.838859 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 29 21:24:00.838859 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 29 21:24:00.838859 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 29 21:24:00.838859 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 21:24:00.838859 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:24:00.838859 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 635/1271] 10.222.0.90 () {56 vars in 1111 bytes} [Mon Aug 29 21:24:00 2022] GET /identity/v3/projects/service => generated 88 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:24:00.843886 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9d2aaa3-2d10-4dfb-aeb5-99c5f99bf822 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:24:00.903591 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:24:00.949628 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9d2aaa3-2d10-4dfb-aeb5-99c5f99bf822 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:24:00.984221 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4ee24f7-88aa-49f8-8aec-2de65bb0c5d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:24:00.984823 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4ee24f7-88aa-49f8-8aec-2de65bb0c5d4 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:24:00.985103 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4ee24f7-88aa-49f8-8aec-2de65bb0c5d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:24:00.985373 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4ee24f7-88aa-49f8-8aec-2de65bb0c5d4 None admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:24:00.986094 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4ee24f7-88aa-49f8-8aec-2de65bb0c5d4 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:24:00.986400 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4ee24f7-88aa-49f8-8aec-2de65bb0c5d4 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:24:00.987814 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4ee24f7-88aa-49f8-8aec-2de65bb0c5d4 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:24:01.001208 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 637/1272] 10.222.0.90 () {56 vars in 1148 bytes} [Mon Aug 29 21:24:00 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:24:01.007277 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31590252-2de4-4b8d-9095-054724a1fdb6 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:24:01.112106 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:24:01.122427 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31590252-2de4-4b8d-9095-054724a1fdb6 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:24:01.165925 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36767f1e-3493-474d-aaf7-af476de9982d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:24:01.166507 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36767f1e-3493-474d-aaf7-af476de9982d None admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:24:01.166852 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36767f1e-3493-474d-aaf7-af476de9982d None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:24:01.167641 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36767f1e-3493-474d-aaf7-af476de9982d None admin] PATH_INFO: `/v3/projects/ec03e112e3d547fbab6faf7b485c3fca/users/b95fa07a7c0549dbaac49406e2639e21/roles/a6e562cdc96c4be691d638990f8403fd` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:24:01.169597 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36767f1e-3493-474d-aaf7-af476de9982d None admin] RBAC: Authorizing `identity:create_grant(project_id=ec03e112e3d547fbab6faf7b485c3fca, user_id=b95fa07a7c0549dbaac49406e2639e21, role_id=a6e562cdc96c4be691d638990f8403fd)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:24:01.171064 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36767f1e-3493-474d-aaf7-af476de9982d None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:24:01.185019 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 636/1273] 10.222.0.90 () {58 vars in 1439 bytes} [Mon Aug 29 21:24:01 2022] PUT /identity/v3/projects/ec03e112e3d547fbab6faf7b485c3fca/users/b95fa07a7c0549dbaac49406e2639e21/roles/a6e562cdc96c4be691d638990f8403fd => generated 0 bytes in 178 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:24:03.181193 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74df0e04-34f2-4914-bc09-ac9b24cb9938 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:24:03.181695 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74df0e04-34f2-4914-bc09-ac9b24cb9938 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:24:03.181695 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74df0e04-34f2-4914-bc09-ac9b24cb9938 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:24:03.183138 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 638/1274] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:24:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:24:03.188199 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f364986b-41da-4fac-8cea-cc02adc3b132 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:24:03.188645 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f364986b-41da-4fac-8cea-cc02adc3b132 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:24:03.188748 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f364986b-41da-4fac-8cea-cc02adc3b132 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:24:03.207170 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-f364986b-41da-4fac-8cea-cc02adc3b132 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:24:03.208361 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-f364986b-41da-4fac-8cea-cc02adc3b132 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:24:03.274935 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 637/1275] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:24:03 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 21:24:03.557119 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74df0e04-34f2-4914-bc09-ac9b24cb9938 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:24:03.656948 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74df0e04-34f2-4914-bc09-ac9b24cb9938 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:24:03.674447 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:24:03.691186 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc7b148f-c2c8-45d3-b596-014cb6aa906f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:24:03.691810 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc7b148f-c2c8-45d3-b596-014cb6aa906f None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:24:03.692086 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc7b148f-c2c8-45d3-b596-014cb6aa906f None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:24:03.692355 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc7b148f-c2c8-45d3-b596-014cb6aa906f None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:24:03.702054 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc7b148f-c2c8-45d3-b596-014cb6aa906f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 21:24:03.706733 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-cc7b148f-c2c8-45d3-b596-014cb6aa906f None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 21:24:03.706733 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:24:03.706733 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:24:03.706733 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:24:03.706733 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:24:03.706733 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:24:03.706733 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:24:03.706733 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:24:03.706733 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:24:03.706733 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:24:03.706733 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:24:03.706733 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:24:03.706733 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 21:24:03.706733 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 21:24:03.706733 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 21:24:03.706733 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 21:24:03.707862 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 21:24:03.707862 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 21:24:03.707862 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 21:24:03.707862 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 21:24:03.707862 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:24:03.707862 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:24:03.707862 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:24:03.707862 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:24:03.707862 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:24:03.707862 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:24:03.707862 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:24:03.707862 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:24:03.707862 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:24:03.707862 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:24:03.707862 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:24:03.707862 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:24:03.709195 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:24:03.709195 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:24:03.709195 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:24:03.709195 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:24:03.709195 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 21:24:03.709195 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 21:24:03.709195 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 21:24:03.709195 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 21:24:03.709195 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 21:24:03.709195 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 21:24:03.709195 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 21:24:03.709195 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:24:03.709195 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 639/1276] 10.222.0.90 () {56 vars in 1096 bytes} [Mon Aug 29 21:24:03 2022] GET /identity/v3/roles/admin => generated 83 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:24:03.714096 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f364986b-41da-4fac-8cea-cc02adc3b132 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:24:03.794230 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:24:03.813363 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f364986b-41da-4fac-8cea-cc02adc3b132 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:24:03.847518 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d580c0fc-4db9-4fb0-9f9f-a71cfb48075e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:24:03.848142 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d580c0fc-4db9-4fb0-9f9f-a71cfb48075e None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:24:03.848423 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d580c0fc-4db9-4fb0-9f9f-a71cfb48075e None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:24:03.848698 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d580c0fc-4db9-4fb0-9f9f-a71cfb48075e None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:24:03.849370 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d580c0fc-4db9-4fb0-9f9f-a71cfb48075e None admin] RBAC: Adding query filter params (name=admin) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:24:03.849668 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d580c0fc-4db9-4fb0-9f9f-a71cfb48075e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:24:03.851068 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d580c0fc-4db9-4fb0-9f9f-a71cfb48075e None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:24:03.860590 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 638/1277] 10.222.0.90 () {56 vars in 1099 bytes} [Mon Aug 29 21:24:03 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:24:03.866335 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc7b148f-c2c8-45d3-b596-014cb6aa906f None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:24:03.971660 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc7b148f-c2c8-45d3-b596-014cb6aa906f None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:24:04.005974 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eb8e8be-e0fd-4ce8-a937-343120b8647e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:24:04.006609 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb8e8be-e0fd-4ce8-a937-343120b8647e None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:24:04.006891 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb8e8be-e0fd-4ce8-a937-343120b8647e None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:24:04.007165 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb8e8be-e0fd-4ce8-a937-343120b8647e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:24:04.017548 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eb8e8be-e0fd-4ce8-a937-343120b8647e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:24:04.019026 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eb8e8be-e0fd-4ce8-a937-343120b8647e None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:24:04.035109 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-1eb8e8be-e0fd-4ce8-a937-343120b8647e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:24:04.035109 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:24:04.035109 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:24:04.035109 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:24:04.035109 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:24:04.035109 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:24:04.035109 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:24:04.035109 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:24:04.035109 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:24:04.035109 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:24:04.035109 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:24:04.035109 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:24:04.035109 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 21:24:04.035109 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 21:24:04.035109 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 21:24:04.035109 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 21:24:04.036247 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:24:04.036247 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:24:04.036247 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:24:04.036247 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:24:04.036247 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:24:04.036247 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:24:04.036247 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:24:04.036247 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:24:04.036247 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:24:04.036247 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:24:04.036247 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:24:04.036247 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:24:04.036247 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:24:04.036247 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:24:04.036247 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:24:04.036247 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:24:04.036247 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 21:24:04.037253 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 21:24:04.037253 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:24:04.037253 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:24:04.037253 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 640/1278] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:24:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:24:04.042861 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d580c0fc-4db9-4fb0-9f9f-a71cfb48075e None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:24:04.142851 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d580c0fc-4db9-4fb0-9f9f-a71cfb48075e None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:24:04.176077 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b9e9cbb-be21-44d1-9022-b7d630294f59 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:24:04.176683 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b9e9cbb-be21-44d1-9022-b7d630294f59 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:24:04.176963 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b9e9cbb-be21-44d1-9022-b7d630294f59 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:24:04.177236 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b9e9cbb-be21-44d1-9022-b7d630294f59 None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:24:04.177903 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b9e9cbb-be21-44d1-9022-b7d630294f59 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:24:04.178216 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b9e9cbb-be21-44d1-9022-b7d630294f59 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:24:04.179570 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b9e9cbb-be21-44d1-9022-b7d630294f59 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:24:04.190936 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 639/1279] 10.222.0.90 () {56 vars in 1109 bytes} [Mon Aug 29 21:24:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:24:04.196597 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eb8e8be-e0fd-4ce8-a937-343120b8647e None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:24:04.229747 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:24:04.299610 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eb8e8be-e0fd-4ce8-a937-343120b8647e None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:24:04.332919 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc0c4cbd-543f-4989-be95-86551fb6dca6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:24:04.333539 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc0c4cbd-543f-4989-be95-86551fb6dca6 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:24:04.333842 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc0c4cbd-543f-4989-be95-86551fb6dca6 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:24:04.334119 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc0c4cbd-543f-4989-be95-86551fb6dca6 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:24:04.341338 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc0c4cbd-543f-4989-be95-86551fb6dca6 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:24:04.343061 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc0c4cbd-543f-4989-be95-86551fb6dca6 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:24:04.353773 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-cc0c4cbd-543f-4989-be95-86551fb6dca6 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Aug 29 21:24:04.353773 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:24:04.353773 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:24:04.353773 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:24:04.353773 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:24:04.353773 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:24:04.353773 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:24:04.353773 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:24:04.353773 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:24:04.353773 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:24:04.353773 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:24:04.353773 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:24:04.353773 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 29 21:24:04.353773 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 29 21:24:04.353773 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 29 21:24:04.353773 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 29 21:24:04.355624 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:24:04.355624 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:24:04.355624 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 29 21:24:04.355624 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 21:24:04.355624 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 29 21:24:04.355624 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 21:24:04.355624 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:24:04.355624 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:24:04.355624 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:24:04.355624 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:24:04.355624 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:24:04.355624 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:24:04.355624 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:24:04.355624 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:24:04.355624 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:24:04.355624 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:24:04.355624 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:24:04.355624 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:24:04.356629 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:24:04.356629 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:24:04.356629 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 29 21:24:04.356629 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 29 21:24:04.356629 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 29 21:24:04.356629 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 29 21:24:04.356629 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 29 21:24:04.356629 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 29 21:24:04.356629 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Aug 29 21:24:04.356629 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:24:04.356629 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 641/1280] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:24:04 2022] GET /identity/v3/users/placement => generated 87 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:24:04.361403 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b9e9cbb-be21-44d1-9022-b7d630294f59 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:24:04.365205 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:24:04.460524 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b9e9cbb-be21-44d1-9022-b7d630294f59 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:24:04.499867 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bc584b1-0b4a-4945-9c18-66f53f060019 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:24:04.500467 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bc584b1-0b4a-4945-9c18-66f53f060019 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:24:04.500748 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bc584b1-0b4a-4945-9c18-66f53f060019 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:24:04.501020 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bc584b1-0b4a-4945-9c18-66f53f060019 None admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:24:04.501749 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bc584b1-0b4a-4945-9c18-66f53f060019 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:24:04.502086 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bc584b1-0b4a-4945-9c18-66f53f060019 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:24:04.503509 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bc584b1-0b4a-4945-9c18-66f53f060019 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:24:04.520201 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 640/1281] 10.222.0.90 () {56 vars in 1143 bytes} [Mon Aug 29 21:24:04 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:24:04.526081 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc0c4cbd-543f-4989-be95-86551fb6dca6 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:24:04.615347 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:24:04.625660 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc0c4cbd-543f-4989-be95-86551fb6dca6 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:24:04.658536 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97d75c6a-731c-4061-9db1-aef6a6df6487 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:24:04.659140 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97d75c6a-731c-4061-9db1-aef6a6df6487 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:24:04.659423 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97d75c6a-731c-4061-9db1-aef6a6df6487 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:24:04.659698 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97d75c6a-731c-4061-9db1-aef6a6df6487 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:24:04.725604 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97d75c6a-731c-4061-9db1-aef6a6df6487 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:24:04.727014 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97d75c6a-731c-4061-9db1-aef6a6df6487 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:24:04.813559 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:24:04.824245 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 642/1282] 10.222.0.90 () {58 vars in 1282 bytes} [Mon Aug 29 21:24:04 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 299 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 29 21:24:04.830110 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bc584b1-0b4a-4945-9c18-66f53f060019 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:24:04.894012 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:24:04.933115 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bc584b1-0b4a-4945-9c18-66f53f060019 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:24:04.965998 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f97453a-72db-4a00-8bf9-30834050a261 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:24:04.966623 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f97453a-72db-4a00-8bf9-30834050a261 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:24:04.967154 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f97453a-72db-4a00-8bf9-30834050a261 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:24:04.967423 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f97453a-72db-4a00-8bf9-30834050a261 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:24:04.977123 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f97453a-72db-4a00-8bf9-30834050a261 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:24:04.978533 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f97453a-72db-4a00-8bf9-30834050a261 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:24:04.993537 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-1f97453a-72db-4a00-8bf9-30834050a261 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:24:04.993537 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:24:04.993537 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:24:04.993537 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:24:04.993537 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:24:04.993537 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:24:04.993537 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:24:04.993537 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:24:04.993537 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:24:04.993537 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:24:04.993537 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:24:04.993537 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:24:04.993537 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 21:24:04.993537 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 21:24:04.993537 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 21:24:04.993537 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 21:24:04.995228 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:24:04.995228 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:24:04.995228 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:24:04.995228 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:24:04.995228 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:24:04.995228 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:24:04.995228 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:24:04.995228 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:24:04.995228 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:24:04.995228 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:24:04.995228 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:24:04.995228 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:24:04.995228 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:24:04.995228 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:24:04.995228 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:24:04.995228 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:24:04.995228 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 21:24:04.996144 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 21:24:04.996144 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 21:24:04.996144 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:24:04.996682 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 641/1283] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:24:04 2022] GET /identity/v3/domains/Default => generated 87 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:24:05.000978 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97d75c6a-731c-4061-9db1-aef6a6df6487 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:24:05.098195 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97d75c6a-731c-4061-9db1-aef6a6df6487 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:24:05.131270 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b7dbc3f-62f0-4b93-8944-83018850fc61 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:24:05.131868 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b7dbc3f-62f0-4b93-8944-83018850fc61 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:24:05.132146 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b7dbc3f-62f0-4b93-8944-83018850fc61 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:24:05.132418 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b7dbc3f-62f0-4b93-8944-83018850fc61 None admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:24:05.133054 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b7dbc3f-62f0-4b93-8944-83018850fc61 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:24:05.133354 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b7dbc3f-62f0-4b93-8944-83018850fc61 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:24:05.134732 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b7dbc3f-62f0-4b93-8944-83018850fc61 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:24:05.145750 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 643/1284] 10.222.0.90 () {56 vars in 1109 bytes} [Mon Aug 29 21:24:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:24:05.151598 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f97453a-72db-4a00-8bf9-30834050a261 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:24:05.250775 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f97453a-72db-4a00-8bf9-30834050a261 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:24:05.263602 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:24:05.283260 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93b74639-0700-4235-b4ca-fbdf57f45486 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:24:05.283877 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93b74639-0700-4235-b4ca-fbdf57f45486 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:24:05.284159 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93b74639-0700-4235-b4ca-fbdf57f45486 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:24:05.284429 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93b74639-0700-4235-b4ca-fbdf57f45486 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:24:05.290898 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93b74639-0700-4235-b4ca-fbdf57f45486 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:24:05.292997 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93b74639-0700-4235-b4ca-fbdf57f45486 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:24:05.303897 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-93b74639-0700-4235-b4ca-fbdf57f45486 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 21:24:05.303897 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:24:05.303897 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:24:05.303897 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:24:05.303897 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:24:05.303897 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:24:05.303897 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:24:05.303897 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:24:05.303897 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:24:05.303897 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:24:05.303897 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:24:05.303897 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:24:05.303897 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 29 21:24:05.303897 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 29 21:24:05.303897 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 29 21:24:05.304976 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 29 21:24:05.304976 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:24:05.304976 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:24:05.304976 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:24:05.304976 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:24:05.304976 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:24:05.304976 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:24:05.304976 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:24:05.304976 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Aug 29 21:24:05.304976 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:24:05.304976 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:24:05.304976 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:24:05.304976 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:24:05.304976 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:24:05.304976 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:24:05.304976 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:24:05.304976 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:24:05.305977 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 29 21:24:05.305977 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 29 21:24:05.305977 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 29 21:24:05.305977 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 29 21:24:05.305977 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 29 21:24:05.305977 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 29 21:24:05.305977 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 21:24:05.305977 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Aug 29 21:24:05.305977 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 642/1285] 10.222.0.90 () {56 vars in 1111 bytes} [Mon Aug 29 21:24:05 2022] GET /identity/v3/projects/service => generated 88 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:24:05.311705 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b7dbc3f-62f0-4b93-8944-83018850fc61 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:24:05.360746 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:24:05.411353 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b7dbc3f-62f0-4b93-8944-83018850fc61 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:24:05.444577 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e26ed0a-c493-4895-aba7-18b098ee7cdf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:24:05.445190 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e26ed0a-c493-4895-aba7-18b098ee7cdf None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:24:05.445488 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e26ed0a-c493-4895-aba7-18b098ee7cdf None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:24:05.445742 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e26ed0a-c493-4895-aba7-18b098ee7cdf None admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:24:05.446466 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e26ed0a-c493-4895-aba7-18b098ee7cdf None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:24:05.446772 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e26ed0a-c493-4895-aba7-18b098ee7cdf None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:24:05.448459 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e26ed0a-c493-4895-aba7-18b098ee7cdf None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:24:05.461586 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 644/1286] 10.222.0.90 () {56 vars in 1148 bytes} [Mon Aug 29 21:24:05 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:24:05.469049 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93b74639-0700-4235-b4ca-fbdf57f45486 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:24:05.567496 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93b74639-0700-4235-b4ca-fbdf57f45486 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:24:05.600687 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-636711a4-4894-4c1d-b613-1ca722ccbe77 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:24:05.601284 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-636711a4-4894-4c1d-b613-1ca722ccbe77 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:24:05.601567 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-636711a4-4894-4c1d-b613-1ca722ccbe77 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:24:05.601867 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-636711a4-4894-4c1d-b613-1ca722ccbe77 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:24:05.602582 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-636711a4-4894-4c1d-b613-1ca722ccbe77 None admin] RBAC: Adding query filter params (role.id=a6e562cdc96c4be691d638990f8403fd, scope.project.id=ec03e112e3d547fbab6faf7b485c3fca, user.id=b95fa07a7c0549dbaac49406e2639e21) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:24:05.602889 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-636711a4-4894-4c1d-b613-1ca722ccbe77 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:24:05.605019 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-636711a4-4894-4c1d-b613-1ca722ccbe77 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:24:05.610783 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 643/1287] 10.222.0.90 () {56 vars in 1374 bytes} [Mon Aug 29 21:24:05 2022] GET /identity/v3/role_assignments?user.id=b95fa07a7c0549dbaac49406e2639e21&scope.project.id=ec03e112e3d547fbab6faf7b485c3fca&role.id=a6e562cdc96c4be691d638990f8403fd => generated 608 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:24:07.525073 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6e70c52-cc98-496c-84a5-0422f0117808 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:24:07.525631 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6e70c52-cc98-496c-84a5-0422f0117808 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:24:07.525815 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6e70c52-cc98-496c-84a5-0422f0117808 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:24:07.527792 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 645/1288] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:24:07 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:24:07.533230 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d617e03f-2daa-4dcd-b279-088a7b0161a9 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:24:07.533663 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d617e03f-2daa-4dcd-b279-088a7b0161a9 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:24:07.533776 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d617e03f-2daa-4dcd-b279-088a7b0161a9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:24:07.552053 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-d617e03f-2daa-4dcd-b279-088a7b0161a9 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:24:07.553230 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-d617e03f-2daa-4dcd-b279-088a7b0161a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:24:07.582706 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:24:07.621630 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 644/1289] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:24:07 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 21:24:07.888012 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6e70c52-cc98-496c-84a5-0422f0117808 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:24:07.987970 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6e70c52-cc98-496c-84a5-0422f0117808 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:24:07.998902 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:24:08.022517 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01610ed7-ef31-4ec7-ba60-076e87b21e60 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:24:08.023147 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01610ed7-ef31-4ec7-ba60-076e87b21e60 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:24:08.023426 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01610ed7-ef31-4ec7-ba60-076e87b21e60 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:24:08.023699 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01610ed7-ef31-4ec7-ba60-076e87b21e60 None admin] PATH_INFO: `/v3/services/placement` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:24:08.024280 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01610ed7-ef31-4ec7-ba60-076e87b21e60 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:24:08.025649 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01610ed7-ef31-4ec7-ba60-076e87b21e60 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:24:08.034559 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-01610ed7-ef31-4ec7-ba60-076e87b21e60 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Aug 29 21:24:08.034559 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:24:08.034559 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 29 21:24:08.034559 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 29 21:24:08.034559 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 29 21:24:08.034559 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 29 21:24:08.034559 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 29 21:24:08.034559 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 21:24:08.034559 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Aug 29 21:24:08.034559 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:24:08.034559 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 29 21:24:08.034559 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:24:08.034559 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:24:08.034559 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:24:08.034559 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:24:08.034559 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:24:08.034559 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:24:08.035857 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:24:08.035857 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:24:08.035857 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:24:08.035857 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:24:08.035857 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:24:08.035857 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:24:08.035857 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 29 21:24:08.035857 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 29 21:24:08.035857 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 29 21:24:08.035857 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 29 21:24:08.035857 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:24:08.035857 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:24:08.035857 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:24:08.035857 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:24:08.035857 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:24:08.035857 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:24:08.036899 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:24:08.036899 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:24:08.036899 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:24:08.036899 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:24:08.036899 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:24:08.036899 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:24:08.036899 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:24:08.036899 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:24:08.036899 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:24:08.036899 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:24:08.036899 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 29 21:24:08.036899 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 21:24:08.036899 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Aug 29 21:24:08.036899 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:24:08.036899 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 646/1290] 10.222.0.90 () {56 vars in 1117 bytes} [Mon Aug 29 21:24:07 2022] GET /identity/v3/services/placement => generated 90 bytes in 148 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:24:08.041735 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d617e03f-2daa-4dcd-b279-088a7b0161a9 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:24:08.139267 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d617e03f-2daa-4dcd-b279-088a7b0161a9 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:24:08.173073 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-661adc1b-82f9-40e5-8a92-2647d688cf4c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:24:08.173682 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-661adc1b-82f9-40e5-8a92-2647d688cf4c None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:24:08.174001 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-661adc1b-82f9-40e5-8a92-2647d688cf4c None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:24:08.174278 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-661adc1b-82f9-40e5-8a92-2647d688cf4c None admin] PATH_INFO: `/v3/services` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:24:08.174909 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-661adc1b-82f9-40e5-8a92-2647d688cf4c None admin] RBAC: Adding query filter params (name=placement) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:24:08.175214 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-661adc1b-82f9-40e5-8a92-2647d688cf4c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:24:08.176582 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-661adc1b-82f9-40e5-8a92-2647d688cf4c None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:24:08.181307 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 645/1291] 10.222.0.90 () {56 vars in 1116 bytes} [Mon Aug 29 21:24:08 2022] GET /identity/v3/services?name=placement => generated 126 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:24:08.187797 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01610ed7-ef31-4ec7-ba60-076e87b21e60 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:24:08.285238 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01610ed7-ef31-4ec7-ba60-076e87b21e60 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:24:08.318171 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89de87e4-6c04-442c-a3d0-1adf931f02e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:24:08.318766 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89de87e4-6c04-442c-a3d0-1adf931f02e7 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:24:08.319045 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89de87e4-6c04-442c-a3d0-1adf931f02e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:24:08.319317 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89de87e4-6c04-442c-a3d0-1adf931f02e7 None admin] PATH_INFO: `/v3/services` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:24:08.319944 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89de87e4-6c04-442c-a3d0-1adf931f02e7 None admin] RBAC: Adding query filter params (type=placement) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:24:08.320244 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89de87e4-6c04-442c-a3d0-1adf931f02e7 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:24:08.321595 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89de87e4-6c04-442c-a3d0-1adf931f02e7 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:24:08.326072 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 647/1292] 10.222.0.90 () {56 vars in 1116 bytes} [Mon Aug 29 21:24:08 2022] GET /identity/v3/services?type=placement => generated 126 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:24:10.234958 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ff75c90-8489-49f1-b109-bb7bfec6d51d None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:24:10.235467 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ff75c90-8489-49f1-b109-bb7bfec6d51d None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:24:10.235467 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ff75c90-8489-49f1-b109-bb7bfec6d51d None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:24:10.236759 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 646/1293] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:24:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:24:10.241770 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3f271d6-fff1-4684-a943-152e0703908d None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:24:10.242161 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3f271d6-fff1-4684-a943-152e0703908d None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:24:10.242331 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3f271d6-fff1-4684-a943-152e0703908d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:24:10.260380 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-a3f271d6-fff1-4684-a943-152e0703908d None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:24:10.261563 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-a3f271d6-fff1-4684-a943-152e0703908d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:24:10.290279 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:24:10.328362 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 648/1294] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:24:10 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 21:24:10.604469 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ff75c90-8489-49f1-b109-bb7bfec6d51d None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:24:10.634691 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:24:10.707776 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ff75c90-8489-49f1-b109-bb7bfec6d51d None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:24:10.741805 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bed4e08b-c9a0-4633-990b-605d63c48fc8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:24:10.742424 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed4e08b-c9a0-4633-990b-605d63c48fc8 None admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:24:10.742702 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed4e08b-c9a0-4633-990b-605d63c48fc8 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:24:10.742974 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed4e08b-c9a0-4633-990b-605d63c48fc8 None admin] PATH_INFO: `/v3/services` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:24:10.743615 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bed4e08b-c9a0-4633-990b-605d63c48fc8 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:24:10.745492 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bed4e08b-c9a0-4633-990b-605d63c48fc8 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:24:10.759329 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 647/1295] 10.222.0.90 () {60 vars in 1141 bytes} [Mon Aug 29 21:24:10 2022] POST /identity/v3/services => generated 246 bytes in 155 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:24:12.650926 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed2fa11-c1d1-4ed5-9a7a-47c7d685f4dd None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:24:12.651496 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed2fa11-c1d1-4ed5-9a7a-47c7d685f4dd None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:24:12.651496 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed2fa11-c1d1-4ed5-9a7a-47c7d685f4dd None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:24:12.652816 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 649/1296] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:24:12 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:24:12.658001 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2cd23f5-b09f-4992-8bea-449a87f8174f None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:24:12.658417 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2cd23f5-b09f-4992-8bea-449a87f8174f None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:24:12.658567 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2cd23f5-b09f-4992-8bea-449a87f8174f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:24:12.676595 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-a2cd23f5-b09f-4992-8bea-449a87f8174f None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:24:12.677800 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-a2cd23f5-b09f-4992-8bea-449a87f8174f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:24:12.749553 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 648/1297] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:24:12 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 21:24:13.021659 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bed2fa11-c1d1-4ed5-9a7a-47c7d685f4dd None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:24:13.123673 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bed2fa11-c1d1-4ed5-9a7a-47c7d685f4dd None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:24:13.156958 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-925a1cca-72e4-4bd1-bc5c-0998840f8a71 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:24:13.157580 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-925a1cca-72e4-4bd1-bc5c-0998840f8a71 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:24:13.157887 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-925a1cca-72e4-4bd1-bc5c-0998840f8a71 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:24:13.158164 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-925a1cca-72e4-4bd1-bc5c-0998840f8a71 None admin] PATH_INFO: `/v3/services/placement` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:24:13.158756 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-925a1cca-72e4-4bd1-bc5c-0998840f8a71 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:24:13.160113 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-925a1cca-72e4-4bd1-bc5c-0998840f8a71 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:24:13.168869 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-925a1cca-72e4-4bd1-bc5c-0998840f8a71 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Aug 29 21:24:13.168869 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:24:13.168869 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 29 21:24:13.168869 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 29 21:24:13.168869 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 29 21:24:13.168869 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 29 21:24:13.168869 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 29 21:24:13.168869 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 21:24:13.168869 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Aug 29 21:24:13.168869 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:24:13.168869 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 29 21:24:13.168869 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:24:13.168869 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:24:13.168869 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:24:13.168869 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:24:13.168869 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:24:13.168869 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:24:13.170099 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:24:13.170099 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:24:13.170099 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:24:13.170099 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:24:13.170099 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:24:13.170099 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:24:13.170099 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 29 21:24:13.170099 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 29 21:24:13.170099 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 29 21:24:13.170099 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 29 21:24:13.170099 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:24:13.170099 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:24:13.170099 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:24:13.170099 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:24:13.170099 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:24:13.170099 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:24:13.171167 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:24:13.171167 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:24:13.171167 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:24:13.171167 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:24:13.171167 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:24:13.171167 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:24:13.171167 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:24:13.171167 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:24:13.171167 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:24:13.171167 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:24:13.171167 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 29 21:24:13.171167 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 21:24:13.171167 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Aug 29 21:24:13.171167 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:24:13.171167 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 650/1298] 10.222.0.90 () {56 vars in 1117 bytes} [Mon Aug 29 21:24:13 2022] GET /identity/v3/services/placement => generated 90 bytes in 149 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 21:24:13.178334 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2cd23f5-b09f-4992-8bea-449a87f8174f None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:24:13.196731 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:24:13.276329 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2cd23f5-b09f-4992-8bea-449a87f8174f None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:24:13.308903 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9e22f96-b346-415a-a9bc-9cc0846faef5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:24:13.309507 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e22f96-b346-415a-a9bc-9cc0846faef5 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:24:13.309814 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e22f96-b346-415a-a9bc-9cc0846faef5 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:24:13.310084 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e22f96-b346-415a-a9bc-9cc0846faef5 None admin] PATH_INFO: `/v3/services` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:24:13.310712 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9e22f96-b346-415a-a9bc-9cc0846faef5 None admin] RBAC: Adding query filter params (name=placement) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:24:13.311008 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9e22f96-b346-415a-a9bc-9cc0846faef5 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:24:13.312357 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9e22f96-b346-415a-a9bc-9cc0846faef5 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:24:13.317374 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 649/1299] 10.222.0.90 () {56 vars in 1116 bytes} [Mon Aug 29 21:24:13 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:24:13.324057 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-925a1cca-72e4-4bd1-bc5c-0998840f8a71 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:24:13.356094 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:24:13.424018 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-925a1cca-72e4-4bd1-bc5c-0998840f8a71 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:24:13.458039 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-defca6c3-6367-4ccb-b866-5a2c959409f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:24:13.458684 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-defca6c3-6367-4ccb-b866-5a2c959409f0 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:24:13.458911 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-defca6c3-6367-4ccb-b866-5a2c959409f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:24:13.459197 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-defca6c3-6367-4ccb-b866-5a2c959409f0 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:24:13.459909 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-defca6c3-6367-4ccb-b866-5a2c959409f0 None admin] RBAC: Adding query filter params (interface=public, service_id=23a38d59fb3d49f58898c1dc82f23cb7, region_id=RegionOne) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:24:13.460215 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-defca6c3-6367-4ccb-b866-5a2c959409f0 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:24:13.461569 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-defca6c3-6367-4ccb-b866-5a2c959409f0 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:24:13.467712 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 651/1300] 10.222.0.90 () {56 vars in 1251 bytes} [Mon Aug 29 21:24:13 2022] GET /identity/v3/endpoints?service_id=23a38d59fb3d49f58898c1dc82f23cb7&interface=public®ion_id=RegionOne => generated 194 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:24:13.473890 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9e22f96-b346-415a-a9bc-9cc0846faef5 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:24:13.558806 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:24:13.577428 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9e22f96-b346-415a-a9bc-9cc0846faef5 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:24:13.611061 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61bbf8fc-0ff7-4e4c-8f24-74c1f83b1f47 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:24:13.611667 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61bbf8fc-0ff7-4e4c-8f24-74c1f83b1f47 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:24:13.611948 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61bbf8fc-0ff7-4e4c-8f24-74c1f83b1f47 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:24:13.612221 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61bbf8fc-0ff7-4e4c-8f24-74c1f83b1f47 None admin] PATH_INFO: `/v3/services` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:24:13.612825 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61bbf8fc-0ff7-4e4c-8f24-74c1f83b1f47 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:24:13.614218 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61bbf8fc-0ff7-4e4c-8f24-74c1f83b1f47 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:24:13.621873 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 650/1301] 10.222.0.90 () {56 vars in 1087 bytes} [Mon Aug 29 21:24:13 2022] GET /identity/v3/services => generated 2191 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:24:15.526105 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddab2c16-c1fb-4540-ade5-b01cd4a6f8a3 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:24:15.526663 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddab2c16-c1fb-4540-ade5-b01cd4a6f8a3 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:24:15.526663 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddab2c16-c1fb-4540-ade5-b01cd4a6f8a3 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:24:15.527987 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 652/1302] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:24:15 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:24:15.533238 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c36d752-8ee2-4f06-b7ab-1097c60f71fd None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:24:15.533668 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c36d752-8ee2-4f06-b7ab-1097c60f71fd None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:24:15.533815 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c36d752-8ee2-4f06-b7ab-1097c60f71fd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:24:15.552866 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-4c36d752-8ee2-4f06-b7ab-1097c60f71fd None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:24:15.554084 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-4c36d752-8ee2-4f06-b7ab-1097c60f71fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:24:15.620484 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 651/1303] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:24:15 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 29 21:24:15.913011 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddab2c16-c1fb-4540-ade5-b01cd4a6f8a3 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:24:16.007817 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:24:16.015557 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddab2c16-c1fb-4540-ade5-b01cd4a6f8a3 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:24:16.048207 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9faf02e0-7a18-426a-8002-58fe44ea9c78 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:24:16.048841 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9faf02e0-7a18-426a-8002-58fe44ea9c78 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:24:16.049126 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9faf02e0-7a18-426a-8002-58fe44ea9c78 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:24:16.049403 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9faf02e0-7a18-426a-8002-58fe44ea9c78 None admin] PATH_INFO: `/v3/services/placement` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:24:16.050035 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9faf02e0-7a18-426a-8002-58fe44ea9c78 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:24:16.051399 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9faf02e0-7a18-426a-8002-58fe44ea9c78 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:24:16.060880 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-9faf02e0-7a18-426a-8002-58fe44ea9c78 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Aug 29 21:24:16.060880 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:24:16.060880 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 29 21:24:16.060880 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 29 21:24:16.060880 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 29 21:24:16.060880 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 29 21:24:16.060880 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 29 21:24:16.060880 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 21:24:16.060880 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Aug 29 21:24:16.060880 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:24:16.060880 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 29 21:24:16.060880 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:24:16.060880 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 21:24:16.060880 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 21:24:16.060880 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 21:24:16.060880 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 21:24:16.060880 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 21:24:16.062118 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 21:24:16.062118 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 21:24:16.062118 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 21:24:16.062118 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 21:24:16.062118 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 21:24:16.062118 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 21:24:16.062118 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 29 21:24:16.062118 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 29 21:24:16.062118 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 29 21:24:16.062118 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 29 21:24:16.062118 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 21:24:16.062118 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 21:24:16.062118 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 21:24:16.062118 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 21:24:16.062118 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 21:24:16.062118 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 21:24:16.063354 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 21:24:16.063354 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Aug 29 21:24:16.063354 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 21:24:16.063354 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Aug 29 21:24:16.063354 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 21:24:16.063354 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 21:24:16.063354 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 21:24:16.063354 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Aug 29 21:24:16.063354 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 21:24:16.063354 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Aug 29 21:24:16.063354 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 29 21:24:16.063354 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 21:24:16.063354 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Aug 29 21:24:16.063354 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Aug 29 21:24:16.063354 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 653/1304] 10.222.0.90 () {56 vars in 1117 bytes} [Mon Aug 29 21:24:15 2022] GET /identity/v3/services/placement => generated 90 bytes in 150 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 29 21:24:16.068145 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c36d752-8ee2-4f06-b7ab-1097c60f71fd None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:24:16.142384 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:24:16.164481 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c36d752-8ee2-4f06-b7ab-1097c60f71fd None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:24:16.198127 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc6c65df-0a91-499d-b8eb-6faef6d98873 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:24:16.198728 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc6c65df-0a91-499d-b8eb-6faef6d98873 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:24:16.199007 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc6c65df-0a91-499d-b8eb-6faef6d98873 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:24:16.199278 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc6c65df-0a91-499d-b8eb-6faef6d98873 None admin] PATH_INFO: `/v3/services` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:24:16.199901 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc6c65df-0a91-499d-b8eb-6faef6d98873 None admin] RBAC: Adding query filter params (name=placement) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:24:16.200201 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc6c65df-0a91-499d-b8eb-6faef6d98873 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:24:16.201555 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc6c65df-0a91-499d-b8eb-6faef6d98873 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:24:16.206537 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 652/1305] 10.222.0.90 () {56 vars in 1116 bytes} [Mon Aug 29 21:24:16 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:24:16.212272 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9faf02e0-7a18-426a-8002-58fe44ea9c78 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:24:16.311491 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9faf02e0-7a18-426a-8002-58fe44ea9c78 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:24:16.345518 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4db9c70-87de-4de0-991c-3d5f4dc8d7a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:24:16.346225 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4db9c70-87de-4de0-991c-3d5f4dc8d7a1 None admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:24:16.346708 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4db9c70-87de-4de0-991c-3d5f4dc8d7a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:24:16.347175 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4db9c70-87de-4de0-991c-3d5f4dc8d7a1 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:24:16.348359 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4db9c70-87de-4de0-991c-3d5f4dc8d7a1 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:24:16.350832 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4db9c70-87de-4de0-991c-3d5f4dc8d7a1 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:24:16.374424 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 654/1306] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:24:16 2022] POST /identity/v3/endpoints => generated 331 bytes in 163 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:24:26.203755 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=30 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:24:26.369529 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=91,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:25:21.017399 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7faf417e-82e4-4048-b06b-ad3aebacb328 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:25:21.017920 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7faf417e-82e4-4048-b06b-ad3aebacb328 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:25:21.018026 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7faf417e-82e4-4048-b06b-ad3aebacb328 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:25:21.022747 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 653/1307] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:25:21 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:25:21.025070 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-190ee46d-a5b5-446e-be48-638c84562737 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:25:21.025481 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-190ee46d-a5b5-446e-be48-638c84562737 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:25:21.025575 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-190ee46d-a5b5-446e-be48-638c84562737 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:25:21.102292 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-190ee46d-a5b5-446e-be48-638c84562737 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:25:21.103424 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-190ee46d-a5b5-446e-be48-638c84562737 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:25:21.235033 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 655/1308] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:25:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 211 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:25:21.540403 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b366c799-e5ab-40e2-8d2c-b45b4cbef3e9 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:25:21.540960 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b366c799-e5ab-40e2-8d2c-b45b4cbef3e9 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:25:21.541387 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b366c799-e5ab-40e2-8d2c-b45b4cbef3e9 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:25:21.543296 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 654/1309] 10.222.0.90 () {54 vars in 965 bytes} [Mon Aug 29 21:25:21 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:25:21.549642 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b03819b-3e16-4bee-8159-edc950b2c22d None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:25:21.549976 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b03819b-3e16-4bee-8159-edc950b2c22d None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:25:21.550264 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b03819b-3e16-4bee-8159-edc950b2c22d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:25:21.647634 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-5b03819b-3e16-4bee-8159-edc950b2c22d None None] MFA Rules not processed for user `0e69b7f0c3574abc9de5fb580d22152a`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:25:21.649097 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-5b03819b-3e16-4bee-8159-edc950b2c22d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:25:21.674313 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:25:21.745992 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 656/1310] 10.222.0.90 () {58 vars in 1062 bytes} [Mon Aug 29 21:25:21 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 197 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:25:21.755450 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b366c799-e5ab-40e2-8d2c-b45b4cbef3e9 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:25:21.758411 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-b366c799-e5ab-40e2-8d2c-b45b4cbef3e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:25:21.895238 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b366c799-e5ab-40e2-8d2c-b45b4cbef3e9 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:25:21.977754 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c90bdebd-7bbf-46c6-9912-98d607ff7e44 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:25:21.978645 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c90bdebd-7bbf-46c6-9912-98d607ff7e44 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:25:21.979085 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c90bdebd-7bbf-46c6-9912-98d607ff7e44 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:25:21.979516 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c90bdebd-7bbf-46c6-9912-98d607ff7e44 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:25:21.981356 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-c90bdebd-7bbf-46c6-9912-98d607ff7e44 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:25:22.051125 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:25:22.111450 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c90bdebd-7bbf-46c6-9912-98d607ff7e44 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:25:22.112788 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c90bdebd-7bbf-46c6-9912-98d607ff7e44 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:25:22.134323 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 655/1311] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:25:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 379 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:25:24.433629 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64797ac0-297d-4dfc-b922-8a65943977e2 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:25:24.434251 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64797ac0-297d-4dfc-b922-8a65943977e2 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:25:24.434251 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64797ac0-297d-4dfc-b922-8a65943977e2 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:25:24.435828 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 657/1312] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:25:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:25:24.441483 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d1a45d4-1655-4395-8b84-7c999b0d166f None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:25:24.441952 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d1a45d4-1655-4395-8b84-7c999b0d166f None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:25:24.442177 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d1a45d4-1655-4395-8b84-7c999b0d166f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:25:24.527451 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-1d1a45d4-1655-4395-8b84-7c999b0d166f None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:25:24.528674 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-1d1a45d4-1655-4395-8b84-7c999b0d166f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:25:24.629450 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 656/1313] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:25:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:25:24.920419 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64797ac0-297d-4dfc-b922-8a65943977e2 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:25:24.922016 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-64797ac0-297d-4dfc-b922-8a65943977e2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:25:25.027576 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64797ac0-297d-4dfc-b922-8a65943977e2 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:25:25.097959 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:25:25.101281 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92c6225f-2126-4661-85fd-8bb618c03c7c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:25:25.102220 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c6225f-2126-4661-85fd-8bb618c03c7c service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:25:25.102569 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c6225f-2126-4661-85fd-8bb618c03c7c service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:25:25.103011 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c6225f-2126-4661-85fd-8bb618c03c7c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:25:25.104847 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-92c6225f-2126-4661-85fd-8bb618c03c7c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:25:25.213135 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92c6225f-2126-4661-85fd-8bb618c03c7c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:25:25.214926 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92c6225f-2126-4661-85fd-8bb618c03c7c service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:25:25.234214 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 658/1314] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:25:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:25:27.458444 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0448e502-3e29-4df8-96df-35f0734fa8f9 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:25:27.459631 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0448e502-3e29-4df8-96df-35f0734fa8f9 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:25:27.459631 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0448e502-3e29-4df8-96df-35f0734fa8f9 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:25:27.460869 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 657/1315] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:25:27 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:25:27.466492 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-674116d9-949b-4519-a2df-13a9296be019 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:25:27.466950 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-674116d9-949b-4519-a2df-13a9296be019 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:25:27.467048 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-674116d9-949b-4519-a2df-13a9296be019 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:25:27.532706 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-674116d9-949b-4519-a2df-13a9296be019 None None] MFA Rules not processed for user `94fd78dbbf6e44308bd6e72458d6361f`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:25:27.533942 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-674116d9-949b-4519-a2df-13a9296be019 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:25:27.586364 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:25:27.631737 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 659/1316] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:25:27 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:25:27.919666 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0448e502-3e29-4df8-96df-35f0734fa8f9 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:25:27.921209 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-0448e502-3e29-4df8-96df-35f0734fa8f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:25:27.931571 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:25:28.027776 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0448e502-3e29-4df8-96df-35f0734fa8f9 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:25:28.103945 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ab2b611-3d96-48dd-86d9-1808194d1bd5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:25:28.104557 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ab2b611-3d96-48dd-86d9-1808194d1bd5 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:25:28.104842 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ab2b611-3d96-48dd-86d9-1808194d1bd5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:25:28.105116 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ab2b611-3d96-48dd-86d9-1808194d1bd5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:25:28.106816 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-9ab2b611-3d96-48dd-86d9-1808194d1bd5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:25:28.178424 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:25:28.226724 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ab2b611-3d96-48dd-86d9-1808194d1bd5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:25:28.228179 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ab2b611-3d96-48dd-86d9-1808194d1bd5 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:25:28.246726 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 658/1317] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:25:27 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 327 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:25:30.676886 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ef82b07-3a66-40ce-a06c-5ca39c138bad None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:25:30.677919 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ef82b07-3a66-40ce-a06c-5ca39c138bad None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:25:30.678023 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ef82b07-3a66-40ce-a06c-5ca39c138bad None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:25:30.679321 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 660/1318] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:25:30 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:25:30.684522 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c250ce5-7d4a-4344-891d-66f19773cd81 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:25:30.684952 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c250ce5-7d4a-4344-891d-66f19773cd81 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:25:30.685074 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c250ce5-7d4a-4344-891d-66f19773cd81 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:25:30.767539 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-7c250ce5-7d4a-4344-891d-66f19773cd81 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:25:30.768463 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-7c250ce5-7d4a-4344-891d-66f19773cd81 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:25:30.866664 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 659/1319] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:25:30 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:25:31.154796 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ef82b07-3a66-40ce-a06c-5ca39c138bad None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:25:31.156341 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-6ef82b07-3a66-40ce-a06c-5ca39c138bad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:25:31.260891 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ef82b07-3a66-40ce-a06c-5ca39c138bad None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:25:31.332743 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2fa97fb-8aa1-4f5c-85c6-ab70b35f4d99 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:25:31.333377 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2fa97fb-8aa1-4f5c-85c6-ab70b35f4d99 service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:25:31.333660 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2fa97fb-8aa1-4f5c-85c6-ab70b35f4d99 service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:25:31.333971 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2fa97fb-8aa1-4f5c-85c6-ab70b35f4d99 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:25:31.335569 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-d2fa97fb-8aa1-4f5c-85c6-ab70b35f4d99 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:25:31.344051 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:25:31.445817 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2fa97fb-8aa1-4f5c-85c6-ab70b35f4d99 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:25:31.447261 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2fa97fb-8aa1-4f5c-85c6-ab70b35f4d99 service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:25:31.465981 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 661/1320] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:25:31 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 312 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:25:34.044251 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16080809-3924-48b0-b7c2-33f3e0c73310 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:25:34.044830 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16080809-3924-48b0-b7c2-33f3e0c73310 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:25:34.044830 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16080809-3924-48b0-b7c2-33f3e0c73310 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:25:34.046217 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 660/1321] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:25:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:25:34.051629 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c551093-e1b7-4c39-8040-82da35af78af None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:25:34.052053 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c551093-e1b7-4c39-8040-82da35af78af None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:25:34.052164 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c551093-e1b7-4c39-8040-82da35af78af None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:25:34.118599 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-4c551093-e1b7-4c39-8040-82da35af78af None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:25:34.119716 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-4c551093-e1b7-4c39-8040-82da35af78af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:25:34.178992 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:25:34.221663 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 662/1322] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:25:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:25:34.508625 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16080809-3924-48b0-b7c2-33f3e0c73310 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:25:34.510360 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-16080809-3924-48b0-b7c2-33f3e0c73310 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:25:34.532057 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:25:34.624553 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16080809-3924-48b0-b7c2-33f3e0c73310 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:25:34.703001 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ede0a74f-d771-417f-8c22-0429b594852a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:25:34.703503 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ede0a74f-d771-417f-8c22-0429b594852a service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:25:34.703783 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ede0a74f-d771-417f-8c22-0429b594852a service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:25:34.704057 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ede0a74f-d771-417f-8c22-0429b594852a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:25:34.705650 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-ede0a74f-d771-417f-8c22-0429b594852a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:25:34.784359 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:25:34.827743 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ede0a74f-d771-417f-8c22-0429b594852a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:25:34.829179 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ede0a74f-d771-417f-8c22-0429b594852a service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:25:34.849314 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 661/1323] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:25:34 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 341 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:25:36.841511 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5329e24-f4a6-4aa7-a4a0-3f98a40f653e None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:25:36.842103 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5329e24-f4a6-4aa7-a4a0-3f98a40f653e None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:25:36.842103 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5329e24-f4a6-4aa7-a4a0-3f98a40f653e None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:25:36.843645 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 663/1324] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:25:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:25:36.848791 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b47659b-f631-4637-8f3c-36aa7f38b0d4 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:25:36.849314 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b47659b-f631-4637-8f3c-36aa7f38b0d4 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:25:36.849424 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b47659b-f631-4637-8f3c-36aa7f38b0d4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:25:36.918855 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-5b47659b-f631-4637-8f3c-36aa7f38b0d4 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:25:36.920067 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-5b47659b-f631-4637-8f3c-36aa7f38b0d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:25:37.019631 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 662/1325] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:25:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:25:37.305697 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5329e24-f4a6-4aa7-a4a0-3f98a40f653e None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:25:37.307279 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-d5329e24-f4a6-4aa7-a4a0-3f98a40f653e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:25:37.424592 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5329e24-f4a6-4aa7-a4a0-3f98a40f653e None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:25:37.502670 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89c0d5e3-3247-4835-ad6a-65ed3927dcef service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:25:37.503284 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89c0d5e3-3247-4835-ad6a-65ed3927dcef service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:25:37.503566 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89c0d5e3-3247-4835-ad6a-65ed3927dcef service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:25:37.503839 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89c0d5e3-3247-4835-ad6a-65ed3927dcef service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:25:37.505434 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-89c0d5e3-3247-4835-ad6a-65ed3927dcef service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:25:37.523920 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:25:37.620154 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89c0d5e3-3247-4835-ad6a-65ed3927dcef service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:25:37.621598 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89c0d5e3-3247-4835-ad6a-65ed3927dcef service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:25:37.640266 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 664/1326] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:25:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 336 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 21:25:39.610065 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2bbef65-6e67-4051-b76c-372f4d2ed552 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:25:39.610630 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2bbef65-6e67-4051-b76c-372f4d2ed552 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:25:39.610630 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2bbef65-6e67-4051-b76c-372f4d2ed552 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:25:39.612026 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 663/1327] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:25:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:25:39.618305 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c74a4ae5-3aba-4498-b52b-a5af396952bd None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:25:39.618829 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c74a4ae5-3aba-4498-b52b-a5af396952bd None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:25:39.618967 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c74a4ae5-3aba-4498-b52b-a5af396952bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:25:39.689167 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-c74a4ae5-3aba-4498-b52b-a5af396952bd None None] MFA Rules not processed for user `94fd78dbbf6e44308bd6e72458d6361f`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:25:39.690390 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-c74a4ae5-3aba-4498-b52b-a5af396952bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:25:39.755986 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:25:39.788379 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 665/1328] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:25:39 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:25:40.081445 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2bbef65-6e67-4051-b76c-372f4d2ed552 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:25:40.083032 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-f2bbef65-6e67-4051-b76c-372f4d2ed552 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:25:40.116201 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:25:40.192679 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2bbef65-6e67-4051-b76c-372f4d2ed552 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:25:40.269556 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4717b2d1-8a77-4f39-93ad-b3199420f940 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:25:40.270115 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4717b2d1-8a77-4f39-93ad-b3199420f940 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:25:40.270392 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4717b2d1-8a77-4f39-93ad-b3199420f940 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:25:40.270661 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4717b2d1-8a77-4f39-93ad-b3199420f940 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:25:40.272249 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-4717b2d1-8a77-4f39-93ad-b3199420f940 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:25:40.351774 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:25:40.383810 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4717b2d1-8a77-4f39-93ad-b3199420f940 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:25:40.385239 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4717b2d1-8a77-4f39-93ad-b3199420f940 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:25:40.404290 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 664/1329] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:25:40 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 323 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:25:42.606038 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9046f071-7730-4572-81c0-f174a0371e66 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:25:42.606507 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9046f071-7730-4572-81c0-f174a0371e66 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:25:42.606624 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9046f071-7730-4572-81c0-f174a0371e66 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:25:42.608596 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 666/1330] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:25:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:25:42.613281 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d0480d2-278f-42eb-840c-4292b1daa7cd None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:25:42.613711 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d0480d2-278f-42eb-840c-4292b1daa7cd None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:25:42.613877 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d0480d2-278f-42eb-840c-4292b1daa7cd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:25:42.693427 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-7d0480d2-278f-42eb-840c-4292b1daa7cd None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:25:42.695460 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-7d0480d2-278f-42eb-840c-4292b1daa7cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:25:42.800567 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 665/1331] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:25:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:25:43.087146 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9046f071-7730-4572-81c0-f174a0371e66 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:25:43.088982 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-9046f071-7730-4572-81c0-f174a0371e66 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:25:43.192407 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9046f071-7730-4572-81c0-f174a0371e66 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:25:43.264352 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3878ebcb-cf01-408f-abbc-5172db26d39b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:25:43.264955 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3878ebcb-cf01-408f-abbc-5172db26d39b service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:25:43.265233 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3878ebcb-cf01-408f-abbc-5172db26d39b service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:25:43.265501 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3878ebcb-cf01-408f-abbc-5172db26d39b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:25:43.267118 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-3878ebcb-cf01-408f-abbc-5172db26d39b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:25:43.293582 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:25:43.376474 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3878ebcb-cf01-408f-abbc-5172db26d39b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:25:43.377942 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3878ebcb-cf01-408f-abbc-5172db26d39b service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:25:43.396199 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 667/1332] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:25:43 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:25:45.341153 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5a34749-93fe-4bb3-a145-fa40ac623e31 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:25:45.341722 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5a34749-93fe-4bb3-a145-fa40ac623e31 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:25:45.341722 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5a34749-93fe-4bb3-a145-fa40ac623e31 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:25:45.343055 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 666/1333] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:25:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:25:45.348197 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-937d446b-ba39-406c-92b7-c83b71412e0f None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:25:45.348626 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-937d446b-ba39-406c-92b7-c83b71412e0f None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:25:45.348773 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-937d446b-ba39-406c-92b7-c83b71412e0f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:25:45.414653 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-937d446b-ba39-406c-92b7-c83b71412e0f None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:25:45.415884 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-937d446b-ba39-406c-92b7-c83b71412e0f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:25:45.489628 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:25:45.512669 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 668/1334] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:25:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:25:45.801492 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5a34749-93fe-4bb3-a145-fa40ac623e31 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:25:45.803067 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-d5a34749-93fe-4bb3-a145-fa40ac623e31 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:25:45.842994 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:25:45.908294 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5a34749-93fe-4bb3-a145-fa40ac623e31 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:25:45.981958 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63d65b0d-f2f8-45e2-9507-665bd74354e9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:25:45.982572 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63d65b0d-f2f8-45e2-9507-665bd74354e9 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:25:45.982852 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63d65b0d-f2f8-45e2-9507-665bd74354e9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:25:45.983123 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63d65b0d-f2f8-45e2-9507-665bd74354e9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:25:45.984736 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-63d65b0d-f2f8-45e2-9507-665bd74354e9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:25:46.068278 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:25:46.095079 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63d65b0d-f2f8-45e2-9507-665bd74354e9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:25:46.096522 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63d65b0d-f2f8-45e2-9507-665bd74354e9 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:25:46.115325 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 667/1335] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:25:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:25:48.337675 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c18f386-deb4-4c01-9e9d-9fa420008747 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:25:48.338528 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c18f386-deb4-4c01-9e9d-9fa420008747 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:25:48.338528 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c18f386-deb4-4c01-9e9d-9fa420008747 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:25:48.339859 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 669/1336] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:25:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:25:48.344929 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae9bf84b-dd59-4673-84ce-f210cc1868c7 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:25:48.345356 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae9bf84b-dd59-4673-84ce-f210cc1868c7 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:25:48.345478 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae9bf84b-dd59-4673-84ce-f210cc1868c7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:25:48.410948 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-ae9bf84b-dd59-4673-84ce-f210cc1868c7 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:25:48.412158 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-ae9bf84b-dd59-4673-84ce-f210cc1868c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:25:48.510774 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 668/1337] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:25:48 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:25:48.806868 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c18f386-deb4-4c01-9e9d-9fa420008747 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:25:48.808417 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-9c18f386-deb4-4c01-9e9d-9fa420008747 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:25:48.912283 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c18f386-deb4-4c01-9e9d-9fa420008747 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:25:48.985881 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-338c6f26-4154-41bb-8a50-acf0ef5f45ee service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:25:48.986490 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-338c6f26-4154-41bb-8a50-acf0ef5f45ee service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:25:48.986772 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-338c6f26-4154-41bb-8a50-acf0ef5f45ee service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:25:48.987048 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-338c6f26-4154-41bb-8a50-acf0ef5f45ee service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:25:48.988638 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-338c6f26-4154-41bb-8a50-acf0ef5f45ee service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:25:49.024321 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:25:49.097503 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-338c6f26-4154-41bb-8a50-acf0ef5f45ee service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:25:49.098901 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-338c6f26-4154-41bb-8a50-acf0ef5f45ee service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:25:49.117048 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 670/1338] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:25:48 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:25:52.998675 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-823838e5-c61d-4066-82cd-7b0663f78235 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:25:52.999204 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-823838e5-c61d-4066-82cd-7b0663f78235 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:25:52.999204 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-823838e5-c61d-4066-82cd-7b0663f78235 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:25:53.000554 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 669/1339] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:25:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:25:53.005886 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa547ad5-3415-48e5-b3ca-c13aa6ef888c None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:25:53.006350 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa547ad5-3415-48e5-b3ca-c13aa6ef888c None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:25:53.006444 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa547ad5-3415-48e5-b3ca-c13aa6ef888c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:25:53.075774 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-aa547ad5-3415-48e5-b3ca-c13aa6ef888c None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:25:53.076976 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-aa547ad5-3415-48e5-b3ca-c13aa6ef888c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:25:53.162267 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:25:53.174987 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 671/1340] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:25:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:25:53.457151 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-823838e5-c61d-4066-82cd-7b0663f78235 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:25:53.458718 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-823838e5-c61d-4066-82cd-7b0663f78235 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:25:53.511281 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:25:53.567755 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-823838e5-c61d-4066-82cd-7b0663f78235 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:25:53.641177 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4003ed0d-92a9-4a96-b6ba-4fd6957be8f3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:25:53.641878 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4003ed0d-92a9-4a96-b6ba-4fd6957be8f3 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:25:53.642306 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4003ed0d-92a9-4a96-b6ba-4fd6957be8f3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:25:53.642583 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4003ed0d-92a9-4a96-b6ba-4fd6957be8f3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:25:53.644165 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-4003ed0d-92a9-4a96-b6ba-4fd6957be8f3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:25:53.738951 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:25:53.759703 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4003ed0d-92a9-4a96-b6ba-4fd6957be8f3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:25:53.761157 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4003ed0d-92a9-4a96-b6ba-4fd6957be8f3 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:25:53.780485 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 670/1341] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:25:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 324 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:25:56.355565 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11bc93e3-db0b-45d4-95b8-5ab94d5f9fb7 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:25:56.355994 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11bc93e3-db0b-45d4-95b8-5ab94d5f9fb7 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:25:56.356104 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11bc93e3-db0b-45d4-95b8-5ab94d5f9fb7 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:25:56.357494 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 672/1342] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:25:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:25:56.362598 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-259c3275-4819-452e-8471-0f664fee6908 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:25:56.362962 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-259c3275-4819-452e-8471-0f664fee6908 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:25:56.363147 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-259c3275-4819-452e-8471-0f664fee6908 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:25:56.434490 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-259c3275-4819-452e-8471-0f664fee6908 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:25:56.435728 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-259c3275-4819-452e-8471-0f664fee6908 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:25:56.549591 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 671/1343] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:25:56 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:25:56.839731 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11bc93e3-db0b-45d4-95b8-5ab94d5f9fb7 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:25:56.841373 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-11bc93e3-db0b-45d4-95b8-5ab94d5f9fb7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:25:56.948319 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11bc93e3-db0b-45d4-95b8-5ab94d5f9fb7 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:25:57.025899 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17d2fa4e-7603-4a08-8c28-b8dc4135ee3f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:25:57.026524 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17d2fa4e-7603-4a08-8c28-b8dc4135ee3f service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:25:57.026806 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17d2fa4e-7603-4a08-8c28-b8dc4135ee3f service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:25:57.027077 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17d2fa4e-7603-4a08-8c28-b8dc4135ee3f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:25:57.028769 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-17d2fa4e-7603-4a08-8c28-b8dc4135ee3f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:25:57.076020 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:25:57.150660 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17d2fa4e-7603-4a08-8c28-b8dc4135ee3f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:25:57.152034 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17d2fa4e-7603-4a08-8c28-b8dc4135ee3f service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:25:57.173154 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 673/1344] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:25:56 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 334 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:26:01.103622 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b34a898c-dd8d-494f-b1ca-9fdb20a89062 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:26:01.104163 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b34a898c-dd8d-494f-b1ca-9fdb20a89062 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:26:01.104163 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b34a898c-dd8d-494f-b1ca-9fdb20a89062 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:26:01.105459 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 672/1345] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:26:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:26:01.110862 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef8a6cc7-514b-4340-930b-b0b79b804903 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:26:01.111298 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef8a6cc7-514b-4340-930b-b0b79b804903 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:26:01.111395 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef8a6cc7-514b-4340-930b-b0b79b804903 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:26:01.191030 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-ef8a6cc7-514b-4340-930b-b0b79b804903 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:26:01.192335 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-ef8a6cc7-514b-4340-930b-b0b79b804903 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:26:01.327215 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:26:01.330848 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 674/1346] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:26:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 221 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:26:01.650454 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c63422a-c744-4eb4-a038-9d9fbc8246fb None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:26:01.650725 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c63422a-c744-4eb4-a038-9d9fbc8246fb None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:26:01.650997 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c63422a-c744-4eb4-a038-9d9fbc8246fb None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:26:01.652332 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 673/1347] 10.222.0.90 () {54 vars in 965 bytes} [Mon Aug 29 21:26:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:26:01.658764 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c7fd854-31f5-460e-b03b-9eeaa42e4fa0 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:26:01.659052 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c7fd854-31f5-460e-b03b-9eeaa42e4fa0 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:26:01.659329 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c7fd854-31f5-460e-b03b-9eeaa42e4fa0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:26:01.742926 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-5c7fd854-31f5-460e-b03b-9eeaa42e4fa0 None None] MFA Rules not processed for user `0e69b7f0c3574abc9de5fb580d22152a`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:26:01.744169 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-5c7fd854-31f5-460e-b03b-9eeaa42e4fa0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:26:01.823620 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 675/1348] 10.222.0.90 () {58 vars in 1062 bytes} [Mon Aug 29 21:26:01 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:26:01.833208 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c63422a-c744-4eb4-a038-9d9fbc8246fb None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:26:01.834889 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-5c63422a-c744-4eb4-a038-9d9fbc8246fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:26:01.900611 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:26:01.948379 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c63422a-c744-4eb4-a038-9d9fbc8246fb None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:26:02.022221 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efed878f-56c1-46cc-bca6-006bfac2eeb3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:26:02.022839 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efed878f-56c1-46cc-bca6-006bfac2eeb3 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:26:02.023119 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efed878f-56c1-46cc-bca6-006bfac2eeb3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:26:02.023391 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efed878f-56c1-46cc-bca6-006bfac2eeb3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:26:02.024994 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-efed878f-56c1-46cc-bca6-006bfac2eeb3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:26:02.137495 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:26:02.141652 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efed878f-56c1-46cc-bca6-006bfac2eeb3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:26:02.143123 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efed878f-56c1-46cc-bca6-006bfac2eeb3 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:26:02.161345 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 674/1349] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:26:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 329 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:26:07.604388 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bf4df02-4121-434a-8c98-dc2e21946023 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:26:07.604826 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bf4df02-4121-434a-8c98-dc2e21946023 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:26:07.604923 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bf4df02-4121-434a-8c98-dc2e21946023 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:26:07.606591 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 676/1350] 10.222.0.90 () {54 vars in 921 bytes} [Mon Aug 29 21:26:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:26:07.611501 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca7d13a9-98ea-4c2a-92fb-2ab43cc57143 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:26:07.611963 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca7d13a9-98ea-4c2a-92fb-2ab43cc57143 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:26:07.612057 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca7d13a9-98ea-4c2a-92fb-2ab43cc57143 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:26:07.727337 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-ca7d13a9-98ea-4c2a-92fb-2ab43cc57143 None None] MFA Rules not processed for user `b95fa07a7c0549dbaac49406e2639e21`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:26:07.728556 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-ca7d13a9-98ea-4c2a-92fb-2ab43cc57143 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:26:07.926759 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 675/1351] 10.222.0.90 () {58 vars in 1018 bytes} [Mon Aug 29 21:26:07 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 316 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:26:11.821814 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=59 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:26:12.149674 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df60e361-f94e-4e79-a03c-2a0cee0d6590 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:26:12.149897 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df60e361-f94e-4e79-a03c-2a0cee0d6590 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:26:12.150258 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df60e361-f94e-4e79-a03c-2a0cee0d6590 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:26:12.151671 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 677/1352] 10.222.0.90 () {54 vars in 921 bytes} [Mon Aug 29 21:26:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:26:12.156488 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d5bdc34-a0fc-46ba-abfc-e2fba4eeb7a9 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:26:12.156963 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d5bdc34-a0fc-46ba-abfc-e2fba4eeb7a9 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:26:12.157047 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d5bdc34-a0fc-46ba-abfc-e2fba4eeb7a9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:26:12.217469 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:26:12.244089 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-4d5bdc34-a0fc-46ba-abfc-e2fba4eeb7a9 None None] MFA Rules not processed for user `b95fa07a7c0549dbaac49406e2639e21`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:26:12.245410 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-4d5bdc34-a0fc-46ba-abfc-e2fba4eeb7a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:26:12.362253 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 676/1353] 10.222.0.90 () {58 vars in 1018 bytes} [Mon Aug 29 21:26:12 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 207 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:26:13.686676 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f52f452-a177-40f7-9c4b-efde3819f8dd None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:26:13.687178 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f52f452-a177-40f7-9c4b-efde3819f8dd None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:26:13.687178 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f52f452-a177-40f7-9c4b-efde3819f8dd None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:26:13.688461 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 678/1354] 10.222.0.90 () {54 vars in 921 bytes} [Mon Aug 29 21:26:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:26:13.694244 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d59e81-0b22-414d-ae24-80dce9dc646b None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:26:13.694655 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d59e81-0b22-414d-ae24-80dce9dc646b None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:26:13.694855 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d59e81-0b22-414d-ae24-80dce9dc646b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:26:13.773457 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-a9d59e81-0b22-414d-ae24-80dce9dc646b None None] MFA Rules not processed for user `b95fa07a7c0549dbaac49406e2639e21`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:26:13.774709 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-a9d59e81-0b22-414d-ae24-80dce9dc646b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:26:13.816869 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:26:13.890033 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 677/1355] 10.222.0.90 () {58 vars in 1018 bytes} [Mon Aug 29 21:26:13 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 197 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 21:26:17.343942 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11821a03-bb50-4de1-b3e2-078e6c7373a8 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:26:17.344487 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11821a03-bb50-4de1-b3e2-078e6c7373a8 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:26:17.344487 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11821a03-bb50-4de1-b3e2-078e6c7373a8 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:26:17.346048 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 679/1356] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:26:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:26:17.351171 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d11fd53a-b816-42ac-8457-a2d5ca0e9810 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:26:17.351587 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d11fd53a-b816-42ac-8457-a2d5ca0e9810 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:26:17.351732 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d11fd53a-b816-42ac-8457-a2d5ca0e9810 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:26:17.436250 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-d11fd53a-b816-42ac-8457-a2d5ca0e9810 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:26:17.437441 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-d11fd53a-b816-42ac-8457-a2d5ca0e9810 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:26:17.582517 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 678/1357] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:26:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 232 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:26:17.895783 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89306b86-fd5b-40e7-853b-aabdecb28c6a None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:26:17.896114 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89306b86-fd5b-40e7-853b-aabdecb28c6a None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:26:17.896457 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89306b86-fd5b-40e7-853b-aabdecb28c6a None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:26:17.897826 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 680/1358] 10.222.0.90 () {54 vars in 962 bytes} [Mon Aug 29 21:26:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:26:17.905749 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a93d5f89-a445-4dcb-9056-9d6f6a202f0e None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:26:17.906269 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a93d5f89-a445-4dcb-9056-9d6f6a202f0e None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:26:17.906691 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a93d5f89-a445-4dcb-9056-9d6f6a202f0e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:26:17.936277 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:26:18.004354 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-a93d5f89-a445-4dcb-9056-9d6f6a202f0e None None] MFA Rules not processed for user `a35b5b7dea7b421da056f063d2436ffb`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:26:18.005504 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-a93d5f89-a445-4dcb-9056-9d6f6a202f0e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:26:18.124847 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 679/1359] 10.222.0.90 () {58 vars in 1059 bytes} [Mon Aug 29 21:26:17 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 220 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:26:18.133569 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89306b86-fd5b-40e7-853b-aabdecb28c6a None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:26:18.135237 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-89306b86-fd5b-40e7-853b-aabdecb28c6a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:26:18.294306 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89306b86-fd5b-40e7-853b-aabdecb28c6a None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:26:18.374638 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:26:18.420144 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be6c6ec4-fe58-43c8-994a-d2568637cb2f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a35b5b7dea7b421da056f063d2436ffb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:26:18.421102 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be6c6ec4-fe58-43c8-994a-d2568637cb2f service nova] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:26:18.421502 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be6c6ec4-fe58-43c8-994a-d2568637cb2f service nova] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:26:18.421957 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be6c6ec4-fe58-43c8-994a-d2568637cb2f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:26:18.423693 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-be6c6ec4-fe58-43c8-994a-d2568637cb2f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:26:18.518009 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be6c6ec4-fe58-43c8-994a-d2568637cb2f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:26:18.519850 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be6c6ec4-fe58-43c8-994a-d2568637cb2f service nova] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:26:18.539669 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 681/1360] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:26:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 406 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:26:20.718839 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee866417-4c64-484a-91ae-9f6a353b0cfa None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:26:20.719345 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee866417-4c64-484a-91ae-9f6a353b0cfa None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:26:20.719345 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee866417-4c64-484a-91ae-9f6a353b0cfa None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:26:20.720826 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 680/1361] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:26:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:26:20.726621 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2030805-6f27-4f65-b256-e790a28d095d None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:26:20.727076 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2030805-6f27-4f65-b256-e790a28d095d None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:26:20.727181 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2030805-6f27-4f65-b256-e790a28d095d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:26:20.806707 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-a2030805-6f27-4f65-b256-e790a28d095d None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:26:20.808485 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-a2030805-6f27-4f65-b256-e790a28d095d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:26:20.851405 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:26:20.925510 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 682/1362] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:26:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 200 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:26:21.219107 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee866417-4c64-484a-91ae-9f6a353b0cfa None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:26:21.220659 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-ee866417-4c64-484a-91ae-9f6a353b0cfa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:26:21.295031 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:26:21.386466 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee866417-4c64-484a-91ae-9f6a353b0cfa None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:26:21.511771 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2411763-dce8-44a5-a025-828583f5f1e8 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a35b5b7dea7b421da056f063d2436ffb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:26:21.512389 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2411763-dce8-44a5-a025-828583f5f1e8 service nova] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:26:21.512862 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2411763-dce8-44a5-a025-828583f5f1e8 service nova] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:26:21.513305 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2411763-dce8-44a5-a025-828583f5f1e8 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:26:21.514948 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-d2411763-dce8-44a5-a025-828583f5f1e8 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:26:21.543969 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:26:21.621777 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2411763-dce8-44a5-a025-828583f5f1e8 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:26:21.623827 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2411763-dce8-44a5-a025-828583f5f1e8 service nova] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:26:21.647580 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 681/1363] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:26:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 429 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:26:23.905172 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da3eaff2-d02c-4e99-b3ad-b2e4783aaff1 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:26:23.905721 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da3eaff2-d02c-4e99-b3ad-b2e4783aaff1 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:26:23.905721 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da3eaff2-d02c-4e99-b3ad-b2e4783aaff1 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:26:23.907104 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 683/1364] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:26:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 29 21:26:23.913296 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f276f684-49ab-44c3-a04f-558f07d05809 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:26:23.913766 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f276f684-49ab-44c3-a04f-558f07d05809 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:26:23.913893 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f276f684-49ab-44c3-a04f-558f07d05809 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:26:23.988346 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-f276f684-49ab-44c3-a04f-558f07d05809 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:26:23.990171 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-f276f684-49ab-44c3-a04f-558f07d05809 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:26:24.094256 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:26:24.096850 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 682/1365] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:26:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:26:24.390176 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6711e099-9b74-4223-915c-c63366ed3250 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:26:24.390537 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6711e099-9b74-4223-915c-c63366ed3250 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:26:24.390869 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6711e099-9b74-4223-915c-c63366ed3250 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:26:24.392261 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 684/1366] 10.222.0.90 () {54 vars in 962 bytes} [Mon Aug 29 21:26:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:26:24.398425 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3f8765e-9c09-4565-bf14-bab3e7ac0ba3 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:26:24.398711 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3f8765e-9c09-4565-bf14-bab3e7ac0ba3 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:26:24.398985 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3f8765e-9c09-4565-bf14-bab3e7ac0ba3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:26:24.481094 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-d3f8765e-9c09-4565-bf14-bab3e7ac0ba3 None None] MFA Rules not processed for user `a35b5b7dea7b421da056f063d2436ffb`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:26:24.482338 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-d3f8765e-9c09-4565-bf14-bab3e7ac0ba3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:26:24.609605 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 683/1367] 10.222.0.90 () {58 vars in 1059 bytes} [Mon Aug 29 21:26:24 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 212 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:26:24.618179 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6711e099-9b74-4223-915c-c63366ed3250 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:26:24.619773 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-6711e099-9b74-4223-915c-c63366ed3250 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:26:24.767270 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6711e099-9b74-4223-915c-c63366ed3250 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:26:24.794614 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:26:24.926964 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-596654aa-7956-4978-b48e-33af45462dc4 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a35b5b7dea7b421da056f063d2436ffb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:26:24.927674 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-596654aa-7956-4978-b48e-33af45462dc4 service nova] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:26:24.927959 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-596654aa-7956-4978-b48e-33af45462dc4 service nova] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:26:24.928149 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-596654aa-7956-4978-b48e-33af45462dc4 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:26:24.929695 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-596654aa-7956-4978-b48e-33af45462dc4 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:26:25.022867 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-596654aa-7956-4978-b48e-33af45462dc4 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:26:25.024728 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-596654aa-7956-4978-b48e-33af45462dc4 service nova] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:26:25.044804 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 685/1368] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:26:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 427 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:26:25.636227 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbc99e2b-bc82-45f7-9f1e-582f7506e185 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:26:25.636761 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbc99e2b-bc82-45f7-9f1e-582f7506e185 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:26:25.636761 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbc99e2b-bc82-45f7-9f1e-582f7506e185 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:26:25.638322 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 684/1369] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:26:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:26:25.643269 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b353fe32-4002-4c10-895d-b32796bbd146 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:26:25.643553 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b353fe32-4002-4c10-895d-b32796bbd146 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:26:25.643823 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b353fe32-4002-4c10-895d-b32796bbd146 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:26:25.679295 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:26:25.717670 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-b353fe32-4002-4c10-895d-b32796bbd146 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:26:25.718956 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-b353fe32-4002-4c10-895d-b32796bbd146 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:26:25.822801 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 686/1370] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:26:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:26:26.412286 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f3da15b-e41e-4624-9f43-a61bcbae48e9 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:26:26.412286 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f3da15b-e41e-4624-9f43-a61bcbae48e9 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:26:26.412286 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f3da15b-e41e-4624-9f43-a61bcbae48e9 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:26:26.415043 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 685/1371] 10.222.0.90 () {54 vars in 964 bytes} [Mon Aug 29 21:26:26 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:26:26.418361 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fe530e8-fa2a-412d-970b-d9ee3c12c66c None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:26:26.418361 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fe530e8-fa2a-412d-970b-d9ee3c12c66c None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:26:26.418361 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fe530e8-fa2a-412d-970b-d9ee3c12c66c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:26:26.497765 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-9fe530e8-fa2a-412d-970b-d9ee3c12c66c None None] MFA Rules not processed for user `0dc226f7ddcb49ecacb51ce57fdfc11b`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:26:26.498996 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-9fe530e8-fa2a-412d-970b-d9ee3c12c66c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:26:26.547509 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:26:26.580872 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 687/1372] 10.222.0.90 () {58 vars in 1061 bytes} [Mon Aug 29 21:26:26 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:26:26.589611 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f3da15b-e41e-4624-9f43-a61bcbae48e9 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:26:26.591168 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-0f3da15b-e41e-4624-9f43-a61bcbae48e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:26:26.644394 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:26:26.702254 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f3da15b-e41e-4624-9f43-a61bcbae48e9 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:26:26.786523 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-167a534f-0667-4bfb-a114-319066f09345 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dc226f7ddcb49ecacb51ce57fdfc11b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:26:26.787263 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-167a534f-0667-4bfb-a114-319066f09345 service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:26:26.787543 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-167a534f-0667-4bfb-a114-319066f09345 service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:26:26.787816 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-167a534f-0667-4bfb-a114-319066f09345 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:26:26.789509 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-167a534f-0667-4bfb-a114-319066f09345 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:26:26.923826 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:26:26.942756 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-167a534f-0667-4bfb-a114-319066f09345 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:26:26.944449 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-167a534f-0667-4bfb-a114-319066f09345 service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:26:26.967912 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 686/1373] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:26:26 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 379 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:26:27.138223 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce4177d4-d2ec-4540-ba18-f73da4d1609e None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:26:27.138566 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce4177d4-d2ec-4540-ba18-f73da4d1609e None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:26:27.139358 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce4177d4-d2ec-4540-ba18-f73da4d1609e None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:26:27.141772 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 688/1374] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:26:27 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:26:27.147138 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02e42128-6e25-445b-ab38-b0c95c4b77dc None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:26:27.147320 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02e42128-6e25-445b-ab38-b0c95c4b77dc None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:26:27.147603 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02e42128-6e25-445b-ab38-b0c95c4b77dc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:26:27.218119 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-02e42128-6e25-445b-ab38-b0c95c4b77dc None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:26:27.219342 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-02e42128-6e25-445b-ab38-b0c95c4b77dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:26:27.334223 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 687/1375] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:26:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:26:27.623669 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce4177d4-d2ec-4540-ba18-f73da4d1609e None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:26:27.625180 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-ce4177d4-d2ec-4540-ba18-f73da4d1609e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:26:27.775353 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce4177d4-d2ec-4540-ba18-f73da4d1609e None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:26:27.826910 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:26:27.891631 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a788fa5-98a7-4343-afe8-32fbf8346170 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a35b5b7dea7b421da056f063d2436ffb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:26:27.892249 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a788fa5-98a7-4343-afe8-32fbf8346170 service nova] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:26:27.892529 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a788fa5-98a7-4343-afe8-32fbf8346170 service nova] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:26:27.893096 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a788fa5-98a7-4343-afe8-32fbf8346170 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:26:27.894478 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-2a788fa5-98a7-4343-afe8-32fbf8346170 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:26:27.985137 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a788fa5-98a7-4343-afe8-32fbf8346170 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:26:27.986773 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a788fa5-98a7-4343-afe8-32fbf8346170 service nova] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:26:28.005134 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 689/1376] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:26:27 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 382 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:26:29.987192 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad015e85-12ef-4e0d-9ed8-9672af712453 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:26:29.987715 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad015e85-12ef-4e0d-9ed8-9672af712453 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:26:29.987811 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad015e85-12ef-4e0d-9ed8-9672af712453 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:26:29.990260 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 688/1377] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:26:29 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:26:29.999147 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-727baf0f-b9b0-48bc-91d9-4f03978d4fb7 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:26:30.000018 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-727baf0f-b9b0-48bc-91d9-4f03978d4fb7 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:26:30.000614 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-727baf0f-b9b0-48bc-91d9-4f03978d4fb7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:26:30.096002 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-727baf0f-b9b0-48bc-91d9-4f03978d4fb7 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:26:30.097188 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-727baf0f-b9b0-48bc-91d9-4f03978d4fb7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:26:30.104854 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:26:30.197835 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 690/1378] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:26:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 202 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:26:30.510402 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad015e85-12ef-4e0d-9ed8-9672af712453 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:26:30.512121 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-ad015e85-12ef-4e0d-9ed8-9672af712453 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:26:30.601161 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:26:30.704385 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad015e85-12ef-4e0d-9ed8-9672af712453 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:26:30.830741 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cdccbbc-f423-48ec-9711-0b2e15b3e48b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a35b5b7dea7b421da056f063d2436ffb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:26:30.831356 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cdccbbc-f423-48ec-9711-0b2e15b3e48b service nova] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:26:30.832181 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cdccbbc-f423-48ec-9711-0b2e15b3e48b service nova] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:26:30.832649 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cdccbbc-f423-48ec-9711-0b2e15b3e48b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:26:30.834289 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-5cdccbbc-f423-48ec-9711-0b2e15b3e48b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:26:30.863650 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:26:30.940110 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cdccbbc-f423-48ec-9711-0b2e15b3e48b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:26:30.942511 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cdccbbc-f423-48ec-9711-0b2e15b3e48b service nova] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:26:30.962724 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 689/1379] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:26:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 453 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:26:32.979214 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7bf83fd-ee6e-4009-8e2c-12fb721963e6 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:26:32.980160 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7bf83fd-ee6e-4009-8e2c-12fb721963e6 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:26:32.980441 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7bf83fd-ee6e-4009-8e2c-12fb721963e6 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:26:32.981804 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 691/1380] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:26:32 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:26:32.987090 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c2231ba-c2d9-4b20-829d-4ed89ea3340e None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:26:32.987520 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c2231ba-c2d9-4b20-829d-4ed89ea3340e None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:26:32.988816 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c2231ba-c2d9-4b20-829d-4ed89ea3340e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:26:33.060567 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-3c2231ba-c2d9-4b20-829d-4ed89ea3340e None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:26:33.062040 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-3c2231ba-c2d9-4b20-829d-4ed89ea3340e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:26:33.160044 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:26:33.169836 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 690/1381] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:26:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:26:33.467286 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7bf83fd-ee6e-4009-8e2c-12fb721963e6 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:26:33.468854 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-c7bf83fd-ee6e-4009-8e2c-12fb721963e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:26:33.630321 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:26:33.641700 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7bf83fd-ee6e-4009-8e2c-12fb721963e6 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:26:33.771775 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6636d9e0-c40b-4dea-8aa4-41a2587e757c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a35b5b7dea7b421da056f063d2436ffb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:26:33.772298 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6636d9e0-c40b-4dea-8aa4-41a2587e757c service nova] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:26:33.772579 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6636d9e0-c40b-4dea-8aa4-41a2587e757c service nova] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:26:33.772853 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6636d9e0-c40b-4dea-8aa4-41a2587e757c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:26:33.774479 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-6636d9e0-c40b-4dea-8aa4-41a2587e757c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:26:33.881545 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6636d9e0-c40b-4dea-8aa4-41a2587e757c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:26:33.883329 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6636d9e0-c40b-4dea-8aa4-41a2587e757c service nova] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:26:33.894212 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:26:33.904132 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 692/1382] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:26:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 437 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:26:35.842567 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3043f7c7-2de4-4b6d-9780-b497e45c8228 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:26:35.843100 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3043f7c7-2de4-4b6d-9780-b497e45c8228 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:26:35.843417 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3043f7c7-2de4-4b6d-9780-b497e45c8228 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:26:35.845232 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 691/1383] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:26:35 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:26:35.850660 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80ea8d3a-dea0-439f-a67a-483a3e4de400 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:26:35.851129 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80ea8d3a-dea0-439f-a67a-483a3e4de400 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:26:35.851231 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80ea8d3a-dea0-439f-a67a-483a3e4de400 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:26:35.918967 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-80ea8d3a-dea0-439f-a67a-483a3e4de400 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:26:35.920145 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-80ea8d3a-dea0-439f-a67a-483a3e4de400 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:26:36.021186 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 693/1384] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:26:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:26:36.308133 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3043f7c7-2de4-4b6d-9780-b497e45c8228 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:26:36.310329 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-3043f7c7-2de4-4b6d-9780-b497e45c8228 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:26:36.482975 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3043f7c7-2de4-4b6d-9780-b497e45c8228 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:26:36.555398 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:26:36.616847 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd5bb062-b797-421b-8f78-5daefa005911 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a35b5b7dea7b421da056f063d2436ffb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:26:36.617614 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd5bb062-b797-421b-8f78-5daefa005911 service nova] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:26:36.618087 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd5bb062-b797-421b-8f78-5daefa005911 service nova] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:26:36.618423 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd5bb062-b797-421b-8f78-5daefa005911 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:26:36.620630 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-bd5bb062-b797-421b-8f78-5daefa005911 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:26:36.716458 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd5bb062-b797-421b-8f78-5daefa005911 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:26:36.718100 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd5bb062-b797-421b-8f78-5daefa005911 service nova] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:26:36.737605 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 692/1385] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:26:36 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 430 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:26:38.675589 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d08ffd0d-335a-4b73-be1e-db45f9081f3c None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:26:38.676454 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d08ffd0d-335a-4b73-be1e-db45f9081f3c None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:26:38.676454 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d08ffd0d-335a-4b73-be1e-db45f9081f3c None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:26:38.677508 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 694/1386] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:26:38 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:26:38.683724 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-387304a6-a142-4eb9-ac60-728c5bd2c8c7 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:26:38.684223 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-387304a6-a142-4eb9-ac60-728c5bd2c8c7 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:26:38.684394 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-387304a6-a142-4eb9-ac60-728c5bd2c8c7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:26:38.756060 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-387304a6-a142-4eb9-ac60-728c5bd2c8c7 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:26:38.757313 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-387304a6-a142-4eb9-ac60-728c5bd2c8c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:26:38.791433 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:26:38.860715 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 693/1387] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:26:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:26:39.154875 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d08ffd0d-335a-4b73-be1e-db45f9081f3c None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:26:39.156405 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-d08ffd0d-335a-4b73-be1e-db45f9081f3c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:26:39.235158 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:26:39.320125 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d08ffd0d-335a-4b73-be1e-db45f9081f3c None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:26:39.454537 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-055b3bff-29a4-4691-a5d8-cf3f271141b1 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a35b5b7dea7b421da056f063d2436ffb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:26:39.455030 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-055b3bff-29a4-4691-a5d8-cf3f271141b1 service nova] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:26:39.455310 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-055b3bff-29a4-4691-a5d8-cf3f271141b1 service nova] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:26:39.455584 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-055b3bff-29a4-4691-a5d8-cf3f271141b1 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:26:39.458315 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-055b3bff-29a4-4691-a5d8-cf3f271141b1 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:26:39.498297 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:26:39.563089 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-055b3bff-29a4-4691-a5d8-cf3f271141b1 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:26:39.564709 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-055b3bff-29a4-4691-a5d8-cf3f271141b1 service nova] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:26:39.583987 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 695/1388] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:26:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 430 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:26:41.532854 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-083f1469-b7f2-4e62-9176-d21550d82aa7 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:26:41.533391 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-083f1469-b7f2-4e62-9176-d21550d82aa7 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:26:41.533391 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-083f1469-b7f2-4e62-9176-d21550d82aa7 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:26:41.535302 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 694/1389] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:26:41 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:26:41.540694 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-140e8826-097c-4325-be09-0facec2eaaaa None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:26:41.541156 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-140e8826-097c-4325-be09-0facec2eaaaa None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:26:41.541261 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-140e8826-097c-4325-be09-0facec2eaaaa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:26:41.609684 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-140e8826-097c-4325-be09-0facec2eaaaa None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:26:41.610931 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-140e8826-097c-4325-be09-0facec2eaaaa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:26:41.709302 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 696/1390] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:26:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:26:42.007862 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-083f1469-b7f2-4e62-9176-d21550d82aa7 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:26:42.009389 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-083f1469-b7f2-4e62-9176-d21550d82aa7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:26:42.050128 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c33020d5-e762-4850-a106-089399d7597c None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:26:42.050482 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c33020d5-e762-4850-a106-089399d7597c None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:26:42.050637 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c33020d5-e762-4850-a106-089399d7597c None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:26:42.052136 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 697/1391] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:26:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 29 21:26:42.057091 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77822a96-a475-4f7b-990a-4379b4bf0d6a None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:26:42.057373 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77822a96-a475-4f7b-990a-4379b4bf0d6a None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:26:42.057647 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77822a96-a475-4f7b-990a-4379b4bf0d6a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:26:42.074356 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:26:42.129444 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-77822a96-a475-4f7b-990a-4379b4bf0d6a None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:26:42.130615 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-77822a96-a475-4f7b-990a-4379b4bf0d6a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:26:42.169204 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-083f1469-b7f2-4e62-9176-d21550d82aa7 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:26:42.186171 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:26:42.234471 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 698/1392] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:26:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 21:26:42.287687 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a44e65d-c65e-455e-b40e-1b148d410b39 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a35b5b7dea7b421da056f063d2436ffb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:26:42.288301 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a44e65d-c65e-455e-b40e-1b148d410b39 service nova] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:26:42.288578 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a44e65d-c65e-455e-b40e-1b148d410b39 service nova] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:26:42.288848 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a44e65d-c65e-455e-b40e-1b148d410b39 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:26:42.290486 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-3a44e65d-c65e-455e-b40e-1b148d410b39 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:26:42.381428 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a44e65d-c65e-455e-b40e-1b148d410b39 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:26:42.382976 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a44e65d-c65e-455e-b40e-1b148d410b39 service nova] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:26:42.402955 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 695/1393] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:26:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 396 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:26:42.547557 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77c1dfbe-2986-4ef1-bc0d-a69129c70bfb None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:26:42.548149 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77c1dfbe-2986-4ef1-bc0d-a69129c70bfb None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:26:42.548149 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77c1dfbe-2986-4ef1-bc0d-a69129c70bfb None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:26:42.549481 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 699/1394] 10.222.0.90 () {54 vars in 967 bytes} [Mon Aug 29 21:26:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:26:42.555185 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a5a8697-2ed4-4011-a982-bd6ba7be5152 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:26:42.555697 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a5a8697-2ed4-4011-a982-bd6ba7be5152 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:26:42.555911 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a5a8697-2ed4-4011-a982-bd6ba7be5152 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:26:42.585396 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:26:42.642330 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-2a5a8697-2ed4-4011-a982-bd6ba7be5152 None None] MFA Rules not processed for user `c63a4eaf48ad43529f0aa328f2ebfd29`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:26:42.643615 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-2a5a8697-2ed4-4011-a982-bd6ba7be5152 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:26:42.722478 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 696/1395] 10.222.0.90 () {58 vars in 1064 bytes} [Mon Aug 29 21:26:42 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:26:42.730185 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77c1dfbe-2986-4ef1-bc0d-a69129c70bfb None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:26:42.731703 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-77c1dfbe-2986-4ef1-bc0d-a69129c70bfb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:26:42.832678 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:26:42.840682 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77c1dfbe-2986-4ef1-bc0d-a69129c70bfb None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:26:42.918257 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27cde9ef-0b6c-47e8-9524-5f46b2cbe061 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c63a4eaf48ad43529f0aa328f2ebfd29', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:26:42.918781 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27cde9ef-0b6c-47e8-9524-5f46b2cbe061 service glance] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:26:42.919060 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27cde9ef-0b6c-47e8-9524-5f46b2cbe061 service glance] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:26:42.919330 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27cde9ef-0b6c-47e8-9524-5f46b2cbe061 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:26:42.920908 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-27cde9ef-0b6c-47e8-9524-5f46b2cbe061 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:26:43.031185 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27cde9ef-0b6c-47e8-9524-5f46b2cbe061 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:26:43.032619 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27cde9ef-0b6c-47e8-9524-5f46b2cbe061 service glance] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:26:43.051579 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 700/1396] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:26:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 322 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:26:43.811278 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13e8d313-a6d0-4cfd-8b99-d6843a232d0d None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:26:43.811768 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13e8d313-a6d0-4cfd-8b99-d6843a232d0d None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:26:43.811900 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13e8d313-a6d0-4cfd-8b99-d6843a232d0d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:26:43.884650 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-13e8d313-a6d0-4cfd-8b99-d6843a232d0d None None] MFA Rules not processed for user `0c0b0da87b124de1be00faec4bbdb404`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:26:43.885933 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-13e8d313-a6d0-4cfd-8b99-d6843a232d0d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:26:43.949309 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:26:43.979341 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 697/1397] 10.222.0.90 () {58 vars in 1009 bytes} [Mon Aug 29 21:26:43 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:26:44.023532 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-197e3428-cf01-4506-a0e5-aeb85c11e0f6 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:26:44.024089 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-197e3428-cf01-4506-a0e5-aeb85c11e0f6 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:26:44.024335 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-197e3428-cf01-4506-a0e5-aeb85c11e0f6 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:26:44.025770 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 701/1398] 10.222.0.90 () {54 vars in 943 bytes} [Mon Aug 29 21:26:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:26:44.032382 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f2eee08-6225-4abe-9045-6f7a515784bf None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:26:44.032741 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f2eee08-6225-4abe-9045-6f7a515784bf None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:26:44.033089 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f2eee08-6225-4abe-9045-6f7a515784bf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:26:44.111164 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-7f2eee08-6225-4abe-9045-6f7a515784bf None None] MFA Rules not processed for user `139bbbde6d884b42b5440727926e2c58`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:26:44.112863 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-7f2eee08-6225-4abe-9045-6f7a515784bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:26:44.227775 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 698/1399] 10.222.0.90 () {58 vars in 1040 bytes} [Mon Aug 29 21:26:44 2022] POST /identity/v3/auth/tokens => generated 3304 bytes in 196 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 21:26:44.237757 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-197e3428-cf01-4506-a0e5-aeb85c11e0f6 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:26:44.239946 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-197e3428-cf01-4506-a0e5-aeb85c11e0f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:26:44.264136 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:26:44.375724 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-197e3428-cf01-4506-a0e5-aeb85c11e0f6 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:26:44.380685 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-813b689a-8978-4330-b1d4-294b44297b84 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:26:44.380975 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-813b689a-8978-4330-b1d4-294b44297b84 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:26:44.381252 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-813b689a-8978-4330-b1d4-294b44297b84 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:26:44.382682 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 699/1400] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:26:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:26:44.387764 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7525d74f-98c1-46ac-8e73-abaf94d7592e None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:26:44.388112 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7525d74f-98c1-46ac-8e73-abaf94d7592e None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:26:44.388461 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7525d74f-98c1-46ac-8e73-abaf94d7592e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:26:44.434266 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:26:44.458500 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-7525d74f-98c1-46ac-8e73-abaf94d7592e None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:26:44.459744 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-7525d74f-98c1-46ac-8e73-abaf94d7592e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:26:44.483710 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-692053a0-970d-47e6-ab24-9ac2cf54569a service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '139bbbde6d884b42b5440727926e2c58', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:26:44.484317 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-692053a0-970d-47e6-ab24-9ac2cf54569a service swift] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:26:44.484595 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-692053a0-970d-47e6-ab24-9ac2cf54569a service swift] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:26:44.484864 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-692053a0-970d-47e6-ab24-9ac2cf54569a service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:26:44.486524 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-692053a0-970d-47e6-ab24-9ac2cf54569a service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:26:44.494918 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:26:44.562776 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 700/1401] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:26:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 21:26:44.573817 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-692053a0-970d-47e6-ab24-9ac2cf54569a service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:26:44.575419 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-692053a0-970d-47e6-ab24-9ac2cf54569a service swift] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:26:44.583682 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 702/1402] 10.222.0.90 () {60 vars in 1384 bytes} [Mon Aug 29 21:26:44 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 347 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 29 21:26:44.868642 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7525d74f-98c1-46ac-8e73-abaf94d7592e None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:26:44.869896 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-7525d74f-98c1-46ac-8e73-abaf94d7592e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:26:45.015889 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7525d74f-98c1-46ac-8e73-abaf94d7592e None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:26:45.035457 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:26:45.197732 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d7a4dea-5b52-4094-b43d-e9e7eb6d445e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a35b5b7dea7b421da056f063d2436ffb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:26:45.198399 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d7a4dea-5b52-4094-b43d-e9e7eb6d445e service nova] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:26:45.198660 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d7a4dea-5b52-4094-b43d-e9e7eb6d445e service nova] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:26:45.198927 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d7a4dea-5b52-4094-b43d-e9e7eb6d445e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:26:45.200590 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-8d7a4dea-5b52-4094-b43d-e9e7eb6d445e service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:26:45.308246 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d7a4dea-5b52-4094-b43d-e9e7eb6d445e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:26:45.310002 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d7a4dea-5b52-4094-b43d-e9e7eb6d445e service nova] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:26:45.331935 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 701/1403] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:26:44 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 464 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:26:47.344812 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccb56f92-c599-4714-9495-8eddfa396f1f None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:26:47.345800 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccb56f92-c599-4714-9495-8eddfa396f1f None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:26:47.346156 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccb56f92-c599-4714-9495-8eddfa396f1f None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:26:47.347701 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 703/1404] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:26:47 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:26:47.362927 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf7b9546-4f48-414c-9b52-f3ff094f895e None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:26:47.362927 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf7b9546-4f48-414c-9b52-f3ff094f895e None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:26:47.362927 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf7b9546-4f48-414c-9b52-f3ff094f895e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:26:47.369123 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:26:47.436690 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-bf7b9546-4f48-414c-9b52-f3ff094f895e None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:26:47.437988 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-bf7b9546-4f48-414c-9b52-f3ff094f895e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:26:47.549931 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 702/1405] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:26:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:26:47.843723 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccb56f92-c599-4714-9495-8eddfa396f1f None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:26:47.845260 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-ccb56f92-c599-4714-9495-8eddfa396f1f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:26:48.065657 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccb56f92-c599-4714-9495-8eddfa396f1f None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:26:48.069258 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:26:48.217431 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a86a71b9-abf5-4505-aa42-e16b63893518 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a35b5b7dea7b421da056f063d2436ffb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:26:48.218095 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a86a71b9-abf5-4505-aa42-e16b63893518 service nova] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:26:48.218496 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a86a71b9-abf5-4505-aa42-e16b63893518 service nova] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:26:48.218892 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a86a71b9-abf5-4505-aa42-e16b63893518 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:26:48.220535 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-a86a71b9-abf5-4505-aa42-e16b63893518 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:26:48.324847 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a86a71b9-abf5-4505-aa42-e16b63893518 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:26:48.326446 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a86a71b9-abf5-4505-aa42-e16b63893518 service nova] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:26:48.349525 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 704/1406] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:26:47 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 506 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:26:50.415417 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb15314d-de82-4091-853c-c2c286e50570 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:26:50.415971 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb15314d-de82-4091-853c-c2c286e50570 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:26:50.415971 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb15314d-de82-4091-853c-c2c286e50570 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:26:50.419645 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 703/1407] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:26:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:26:50.426014 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4662154b-5495-4b1f-9e5d-395e1af11428 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:26:50.426502 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4662154b-5495-4b1f-9e5d-395e1af11428 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:26:50.426711 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4662154b-5495-4b1f-9e5d-395e1af11428 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:26:50.436688 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:26:50.506705 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-4662154b-5495-4b1f-9e5d-395e1af11428 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:26:50.508416 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-4662154b-5495-4b1f-9e5d-395e1af11428 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:26:50.617191 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 705/1408] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:26:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 193 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:26:50.909887 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb15314d-de82-4091-853c-c2c286e50570 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:26:50.911314 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-eb15314d-de82-4091-853c-c2c286e50570 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:26:51.016474 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:26:51.090535 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb15314d-de82-4091-853c-c2c286e50570 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:26:51.226821 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-327ca396-d1ca-4a69-9a8d-ed2f12829e26 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a35b5b7dea7b421da056f063d2436ffb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:26:51.228172 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-327ca396-d1ca-4a69-9a8d-ed2f12829e26 service nova] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:26:51.228531 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-327ca396-d1ca-4a69-9a8d-ed2f12829e26 service nova] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:26:51.228832 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-327ca396-d1ca-4a69-9a8d-ed2f12829e26 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:26:51.230630 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-327ca396-d1ca-4a69-9a8d-ed2f12829e26 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:26:51.287371 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:26:51.337705 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-327ca396-d1ca-4a69-9a8d-ed2f12829e26 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:26:51.339532 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-327ca396-d1ca-4a69-9a8d-ed2f12829e26 service nova] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:26:51.358978 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 704/1409] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:26:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 451 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:27:00.615013 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=62 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:27:01.356590 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=23 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:27:07.510138 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-184d2109-4928-4a14-83e9-494936a11aec None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:27:07.510730 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-184d2109-4928-4a14-83e9-494936a11aec None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:27:07.510730 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-184d2109-4928-4a14-83e9-494936a11aec None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:27:07.512054 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 706/1410] 10.222.0.90 () {54 vars in 918 bytes} [Mon Aug 29 21:27:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:27:07.517741 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8d26c1e-4875-4aa9-88af-acd5c1483be6 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:27:07.518328 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8d26c1e-4875-4aa9-88af-acd5c1483be6 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:27:07.518440 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8d26c1e-4875-4aa9-88af-acd5c1483be6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:27:07.595915 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-e8d26c1e-4875-4aa9-88af-acd5c1483be6 None None] MFA Rules not processed for user `b95fa07a7c0549dbaac49406e2639e21`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:27:07.597245 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-e8d26c1e-4875-4aa9-88af-acd5c1483be6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:27:07.703509 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 705/1411] 10.222.0.90 () {58 vars in 1015 bytes} [Mon Aug 29 21:27:07 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:27:07.718561 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00accb5e-f0f9-4395-900d-67f9042b8625 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:27:07.718855 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00accb5e-f0f9-4395-900d-67f9042b8625 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:27:07.719136 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00accb5e-f0f9-4395-900d-67f9042b8625 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:27:07.720540 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 707/1412] 10.222.0.90 () {54 vars in 918 bytes} [Mon Aug 29 21:27:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:27:07.725271 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cba1b35-3b91-4dc8-9718-31b90952df90 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:27:07.725559 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cba1b35-3b91-4dc8-9718-31b90952df90 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:27:07.725866 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cba1b35-3b91-4dc8-9718-31b90952df90 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:27:07.800096 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:27:07.803359 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-9cba1b35-3b91-4dc8-9718-31b90952df90 None None] MFA Rules not processed for user `a35b5b7dea7b421da056f063d2436ffb`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:27:07.804963 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-9cba1b35-3b91-4dc8-9718-31b90952df90 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:27:07.923289 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 706/1413] 10.222.0.90 () {58 vars in 1015 bytes} [Mon Aug 29 21:27:07 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 199 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 21:27:10.326244 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab24ad93-ebe3-46a7-81a6-93ccdc63b6f8 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:27:10.326803 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab24ad93-ebe3-46a7-81a6-93ccdc63b6f8 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:27:10.326803 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab24ad93-ebe3-46a7-81a6-93ccdc63b6f8 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:27:10.328163 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 708/1414] 127.0.0.1 () {54 vars in 917 bytes} [Mon Aug 29 21:27:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:27:10.334012 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-305448d3-ecd4-4f02-9786-ce099a1607f0 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:27:10.334443 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-305448d3-ecd4-4f02-9786-ce099a1607f0 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:27:10.334550 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-305448d3-ecd4-4f02-9786-ce099a1607f0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:27:10.402412 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-305448d3-ecd4-4f02-9786-ce099a1607f0 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:27:10.403639 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-305448d3-ecd4-4f02-9786-ce099a1607f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:27:10.461578 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:27:10.504877 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 707/1415] 127.0.0.1 () {58 vars in 1014 bytes} [Mon Aug 29 21:27:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:27:10.797101 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab24ad93-ebe3-46a7-81a6-93ccdc63b6f8 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:27:10.799706 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-ab24ad93-ebe3-46a7-81a6-93ccdc63b6f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:27:10.946093 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab24ad93-ebe3-46a7-81a6-93ccdc63b6f8 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:27:11.019313 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:27:11.059472 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9333274c-3728-45a5-bbf9-01345b329aed service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a35b5b7dea7b421da056f063d2436ffb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:27:11.060086 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9333274c-3728-45a5-bbf9-01345b329aed service nova] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:27:11.060363 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9333274c-3728-45a5-bbf9-01345b329aed service nova] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:27:11.060634 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9333274c-3728-45a5-bbf9-01345b329aed service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:27:11.062288 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-9333274c-3728-45a5-bbf9-01345b329aed service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:27:11.152373 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9333274c-3728-45a5-bbf9-01345b329aed service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:27:11.153921 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9333274c-3728-45a5-bbf9-01345b329aed service nova] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:27:11.172022 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 709/1416] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:27:10 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 375 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:27:12.084126 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b41e9bf2-a1f2-4d42-9c7e-e4adc939a359 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:27:12.084693 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b41e9bf2-a1f2-4d42-9c7e-e4adc939a359 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:27:12.084693 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b41e9bf2-a1f2-4d42-9c7e-e4adc939a359 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:27:12.086334 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 708/1417] 127.0.0.1 () {54 vars in 901 bytes} [Mon Aug 29 21:27:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:27:12.091314 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e0a61f1-0310-4106-aa7f-48e82cdc383f None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:27:12.091743 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e0a61f1-0310-4106-aa7f-48e82cdc383f None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:27:12.091846 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e0a61f1-0310-4106-aa7f-48e82cdc383f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:27:12.169948 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-3e0a61f1-0310-4106-aa7f-48e82cdc383f None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:27:12.171081 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-3e0a61f1-0310-4106-aa7f-48e82cdc383f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:27:12.199024 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:27:12.270467 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 710/1418] 127.0.0.1 () {58 vars in 998 bytes} [Mon Aug 29 21:27:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:27:12.422009 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b41e9bf2-a1f2-4d42-9c7e-e4adc939a359 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:27:12.424063 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-b41e9bf2-a1f2-4d42-9c7e-e4adc939a359 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:27:12.570931 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b41e9bf2-a1f2-4d42-9c7e-e4adc939a359 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:27:12.612474 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:27:12.685804 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1731ff85-2b5d-45d1-a4e2-56c2a68a020a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a35b5b7dea7b421da056f063d2436ffb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:27:12.686347 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1731ff85-2b5d-45d1-a4e2-56c2a68a020a service nova] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:27:12.686629 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1731ff85-2b5d-45d1-a4e2-56c2a68a020a service nova] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:27:12.686904 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1731ff85-2b5d-45d1-a4e2-56c2a68a020a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:27:12.688521 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-1731ff85-2b5d-45d1-a4e2-56c2a68a020a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:27:12.782990 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1731ff85-2b5d-45d1-a4e2-56c2a68a020a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:27:12.784752 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1731ff85-2b5d-45d1-a4e2-56c2a68a020a service nova] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:27:12.802996 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 709/1419] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:27:12 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 382 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:27:13.712176 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-062b4abd-be0e-48c8-a855-d881ee1e8dea None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:27:13.712748 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-062b4abd-be0e-48c8-a855-d881ee1e8dea None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:27:13.712748 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-062b4abd-be0e-48c8-a855-d881ee1e8dea None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:27:13.714147 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 711/1420] 127.0.0.1 () {54 vars in 901 bytes} [Mon Aug 29 21:27:13 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:27:13.719097 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b23daf3d-169a-4f46-aad8-6537f909fdfb None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:27:13.719511 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b23daf3d-169a-4f46-aad8-6537f909fdfb None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:27:13.719636 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b23daf3d-169a-4f46-aad8-6537f909fdfb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:27:13.775270 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:27:13.797754 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-b23daf3d-169a-4f46-aad8-6537f909fdfb None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:27:13.799057 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-b23daf3d-169a-4f46-aad8-6537f909fdfb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:27:13.899230 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 710/1421] 127.0.0.1 () {58 vars in 998 bytes} [Mon Aug 29 21:27:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:27:14.053247 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-062b4abd-be0e-48c8-a855-d881ee1e8dea None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:27:14.054846 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-062b4abd-be0e-48c8-a855-d881ee1e8dea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:27:14.207581 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-062b4abd-be0e-48c8-a855-d881ee1e8dea None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:27:14.219954 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:27:14.322038 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e82a2f61-9339-4be5-930b-f2c29f05f8e3 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a35b5b7dea7b421da056f063d2436ffb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:27:14.322558 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e82a2f61-9339-4be5-930b-f2c29f05f8e3 service nova] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:27:14.322838 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e82a2f61-9339-4be5-930b-f2c29f05f8e3 service nova] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:27:14.323110 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e82a2f61-9339-4be5-930b-f2c29f05f8e3 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:27:14.324725 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-e82a2f61-9339-4be5-930b-f2c29f05f8e3 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:27:14.414883 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e82a2f61-9339-4be5-930b-f2c29f05f8e3 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:27:14.416495 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e82a2f61-9339-4be5-930b-f2c29f05f8e3 service nova] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:27:14.435108 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 712/1422] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:27:14 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 382 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:27:15.326501 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb34262-22c5-4444-82f2-b9fcd047a09b None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:27:15.327332 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb34262-22c5-4444-82f2-b9fcd047a09b None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:27:15.327630 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb34262-22c5-4444-82f2-b9fcd047a09b None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:27:15.328956 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 711/1423] 127.0.0.1 () {54 vars in 901 bytes} [Mon Aug 29 21:27:15 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:27:15.334830 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18969852-a91f-4c62-b160-ac6b37a70773 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:27:15.335339 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18969852-a91f-4c62-b160-ac6b37a70773 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:27:15.335436 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18969852-a91f-4c62-b160-ac6b37a70773 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:27:15.367116 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:27:15.442959 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-18969852-a91f-4c62-b160-ac6b37a70773 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:27:15.444300 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-18969852-a91f-4c62-b160-ac6b37a70773 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:27:15.548540 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 713/1424] 127.0.0.1 () {58 vars in 998 bytes} [Mon Aug 29 21:27:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 215 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:27:15.702803 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fb34262-22c5-4444-82f2-b9fcd047a09b None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:27:15.704247 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-8fb34262-22c5-4444-82f2-b9fcd047a09b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:27:15.842745 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:27:15.882954 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fb34262-22c5-4444-82f2-b9fcd047a09b None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:27:16.007197 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58f0a591-90fd-4997-b456-281df3e34b25 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a35b5b7dea7b421da056f063d2436ffb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:27:16.007725 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58f0a591-90fd-4997-b456-281df3e34b25 service nova] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:27:16.008006 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58f0a591-90fd-4997-b456-281df3e34b25 service nova] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:27:16.008281 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58f0a591-90fd-4997-b456-281df3e34b25 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:27:16.009935 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-58f0a591-90fd-4997-b456-281df3e34b25 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:27:16.102515 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:27:16.106679 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58f0a591-90fd-4997-b456-281df3e34b25 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:27:16.108701 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58f0a591-90fd-4997-b456-281df3e34b25 service nova] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:27:16.136146 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 712/1425] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:27:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 434 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:27:17.053630 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6545e55c-8f89-4b34-8f00-f82468fec49b None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:27:17.054242 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6545e55c-8f89-4b34-8f00-f82468fec49b None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:27:17.054242 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6545e55c-8f89-4b34-8f00-f82468fec49b None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:27:17.055577 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 714/1426] 127.0.0.1 () {54 vars in 901 bytes} [Mon Aug 29 21:27:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:27:17.060735 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3653571e-7f16-4b22-bbd8-05cffc829545 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:27:17.061158 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3653571e-7f16-4b22-bbd8-05cffc829545 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:27:17.061322 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3653571e-7f16-4b22-bbd8-05cffc829545 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:27:17.137368 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-3653571e-7f16-4b22-bbd8-05cffc829545 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:27:17.138539 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-3653571e-7f16-4b22-bbd8-05cffc829545 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:27:17.234636 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 713/1427] 127.0.0.1 () {58 vars in 998 bytes} [Mon Aug 29 21:27:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:27:17.385421 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6545e55c-8f89-4b34-8f00-f82468fec49b None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:27:17.386975 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-6545e55c-8f89-4b34-8f00-f82468fec49b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:27:17.484242 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:27:17.545502 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6545e55c-8f89-4b34-8f00-f82468fec49b None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:27:17.657617 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b4cfae7-0b22-460a-a4ae-f772848d6890 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a35b5b7dea7b421da056f063d2436ffb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:27:17.658854 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b4cfae7-0b22-460a-a4ae-f772848d6890 service nova] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:27:17.659136 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b4cfae7-0b22-460a-a4ae-f772848d6890 service nova] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:27:17.659412 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b4cfae7-0b22-460a-a4ae-f772848d6890 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:27:17.661037 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-7b4cfae7-0b22-460a-a4ae-f772848d6890 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:27:17.708610 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:27:17.753504 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b4cfae7-0b22-460a-a4ae-f772848d6890 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:27:17.755176 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b4cfae7-0b22-460a-a4ae-f772848d6890 service nova] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:27:17.774542 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 715/1428] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:27:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 390 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:27:18.669780 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-435b37b2-2e95-42c6-bc7b-ee4a7148d2ec None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:27:18.670375 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-435b37b2-2e95-42c6-bc7b-ee4a7148d2ec None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:27:18.670375 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-435b37b2-2e95-42c6-bc7b-ee4a7148d2ec None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:27:18.671713 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 714/1429] 127.0.0.1 () {54 vars in 901 bytes} [Mon Aug 29 21:27:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:27:18.676839 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bb4fc80-d6f4-4f38-817a-b20a750fd522 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:27:18.677298 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bb4fc80-d6f4-4f38-817a-b20a750fd522 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:27:18.677414 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bb4fc80-d6f4-4f38-817a-b20a750fd522 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:27:18.757891 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-8bb4fc80-d6f4-4f38-817a-b20a750fd522 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:27:18.759447 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-8bb4fc80-d6f4-4f38-817a-b20a750fd522 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:27:18.863826 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 716/1430] 127.0.0.1 () {58 vars in 998 bytes} [Mon Aug 29 21:27:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:27:19.022739 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-435b37b2-2e95-42c6-bc7b-ee4a7148d2ec None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:27:19.024430 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-435b37b2-2e95-42c6-bc7b-ee4a7148d2ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:27:19.075400 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:27:19.183913 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-435b37b2-2e95-42c6-bc7b-ee4a7148d2ec None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:27:19.308511 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e376440f-aa3b-4d34-9170-1840b39728ca service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a35b5b7dea7b421da056f063d2436ffb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:27:19.309121 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e376440f-aa3b-4d34-9170-1840b39728ca service nova] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:27:19.309400 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e376440f-aa3b-4d34-9170-1840b39728ca service nova] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:27:19.309674 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e376440f-aa3b-4d34-9170-1840b39728ca service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:27:19.311548 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-e376440f-aa3b-4d34-9170-1840b39728ca service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:27:19.319985 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:27:19.420431 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e376440f-aa3b-4d34-9170-1840b39728ca service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:27:19.422065 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e376440f-aa3b-4d34-9170-1840b39728ca service nova] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:27:19.441606 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 715/1431] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:27:19 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 419 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:27:20.305897 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c49d5fb9-a999-42ff-b180-8bd3126b279d None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:27:20.306519 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c49d5fb9-a999-42ff-b180-8bd3126b279d None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:27:20.306519 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c49d5fb9-a999-42ff-b180-8bd3126b279d None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:27:20.308435 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 717/1432] 127.0.0.1 () {54 vars in 901 bytes} [Mon Aug 29 21:27:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:27:20.313020 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e963949-d72b-45ef-bcbd-f76f1ed0433b None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:27:20.313487 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e963949-d72b-45ef-bcbd-f76f1ed0433b None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:27:20.313587 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e963949-d72b-45ef-bcbd-f76f1ed0433b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:27:20.395180 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-8e963949-d72b-45ef-bcbd-f76f1ed0433b None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:27:20.396315 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-8e963949-d72b-45ef-bcbd-f76f1ed0433b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:27:20.474712 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:27:20.507716 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 716/1433] 127.0.0.1 () {58 vars in 998 bytes} [Mon Aug 29 21:27:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 196 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:27:20.663754 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c49d5fb9-a999-42ff-b180-8bd3126b279d None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:27:20.665233 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-c49d5fb9-a999-42ff-b180-8bd3126b279d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:27:20.683530 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:27:20.850242 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c49d5fb9-a999-42ff-b180-8bd3126b279d None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:27:20.945516 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:27:20.973336 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69790ebd-cedc-40d0-96ff-b65890e97229 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a35b5b7dea7b421da056f063d2436ffb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:27:20.974017 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69790ebd-cedc-40d0-96ff-b65890e97229 service nova] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:27:20.974300 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69790ebd-cedc-40d0-96ff-b65890e97229 service nova] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:27:20.974578 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69790ebd-cedc-40d0-96ff-b65890e97229 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:27:20.976211 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-69790ebd-cedc-40d0-96ff-b65890e97229 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:27:21.068994 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69790ebd-cedc-40d0-96ff-b65890e97229 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:27:21.070653 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69790ebd-cedc-40d0-96ff-b65890e97229 service nova] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:27:21.091407 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 718/1434] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:27:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 428 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:27:21.917585 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f850d6d1-643e-4be7-ad13-530044e720cb None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:27:21.918180 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f850d6d1-643e-4be7-ad13-530044e720cb None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:27:21.918310 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f850d6d1-643e-4be7-ad13-530044e720cb None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:27:21.919654 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 717/1435] 127.0.0.1 () {54 vars in 901 bytes} [Mon Aug 29 21:27:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:27:21.925225 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f3cf479-ae6f-43ea-89ac-fdb13eb9f103 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:27:21.925683 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f3cf479-ae6f-43ea-89ac-fdb13eb9f103 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:27:21.925817 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f3cf479-ae6f-43ea-89ac-fdb13eb9f103 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:27:22.005682 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-9f3cf479-ae6f-43ea-89ac-fdb13eb9f103 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:27:22.006949 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-9f3cf479-ae6f-43ea-89ac-fdb13eb9f103 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:27:22.048164 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:27:22.103825 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 719/1436] 127.0.0.1 () {58 vars in 998 bytes} [Mon Aug 29 21:27:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:27:22.256609 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f850d6d1-643e-4be7-ad13-530044e720cb None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:27:22.258203 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-f850d6d1-643e-4be7-ad13-530044e720cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:27:22.405487 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f850d6d1-643e-4be7-ad13-530044e720cb None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:27:22.459411 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:27:22.519788 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ab19515-bba8-4362-a039-6b4ceeab1ad6 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a35b5b7dea7b421da056f063d2436ffb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:27:22.520401 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ab19515-bba8-4362-a039-6b4ceeab1ad6 service nova] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:27:22.520683 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ab19515-bba8-4362-a039-6b4ceeab1ad6 service nova] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:27:22.520959 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ab19515-bba8-4362-a039-6b4ceeab1ad6 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:27:22.522605 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-1ab19515-bba8-4362-a039-6b4ceeab1ad6 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:27:22.614178 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ab19515-bba8-4362-a039-6b4ceeab1ad6 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:27:22.615787 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ab19515-bba8-4362-a039-6b4ceeab1ad6 service nova] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:27:22.634942 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 718/1437] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:27:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 379 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:27:23.446154 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc746c4b-a53f-4d1e-b428-91520440fdf1 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:27:23.446744 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc746c4b-a53f-4d1e-b428-91520440fdf1 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:27:23.446744 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc746c4b-a53f-4d1e-b428-91520440fdf1 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:27:23.448105 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 720/1438] 127.0.0.1 () {54 vars in 901 bytes} [Mon Aug 29 21:27:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:27:23.453115 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfc13112-9bb3-4859-ba12-989f8cb1798d None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:27:23.453541 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfc13112-9bb3-4859-ba12-989f8cb1798d None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:27:23.453657 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfc13112-9bb3-4859-ba12-989f8cb1798d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:27:23.532851 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-cfc13112-9bb3-4859-ba12-989f8cb1798d None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:27:23.534104 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-cfc13112-9bb3-4859-ba12-989f8cb1798d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:27:23.536213 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:27:23.635121 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 719/1439] 127.0.0.1 () {58 vars in 998 bytes} [Mon Aug 29 21:27:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:27:23.789315 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc746c4b-a53f-4d1e-b428-91520440fdf1 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:27:23.790963 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-bc746c4b-a53f-4d1e-b428-91520440fdf1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:27:23.944984 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc746c4b-a53f-4d1e-b428-91520440fdf1 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:27:23.971765 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:27:24.063263 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a79fa382-d8b7-4fd7-9a7f-4cfa292ff43f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a35b5b7dea7b421da056f063d2436ffb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:27:24.063796 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a79fa382-d8b7-4fd7-9a7f-4cfa292ff43f service nova] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:27:24.064141 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a79fa382-d8b7-4fd7-9a7f-4cfa292ff43f service nova] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:27:24.064928 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a79fa382-d8b7-4fd7-9a7f-4cfa292ff43f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:27:24.066659 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-a79fa382-d8b7-4fd7-9a7f-4cfa292ff43f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:27:24.164139 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a79fa382-d8b7-4fd7-9a7f-4cfa292ff43f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:27:24.166028 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a79fa382-d8b7-4fd7-9a7f-4cfa292ff43f service nova] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:27:24.185265 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 721/1440] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:27:23 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 396 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:27:26.092369 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-541e2ff6-1074-4bbf-b57e-68c5d9093e5b None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:27:26.092947 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-541e2ff6-1074-4bbf-b57e-68c5d9093e5b None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:27:26.092947 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-541e2ff6-1074-4bbf-b57e-68c5d9093e5b None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:27:26.094303 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 720/1441] 127.0.0.1 () {54 vars in 917 bytes} [Mon Aug 29 21:27:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:27:26.100335 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-497ce4dd-0ed3-4a5d-bc09-9629c0118fb8 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:27:26.100781 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-497ce4dd-0ed3-4a5d-bc09-9629c0118fb8 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:27:26.100916 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-497ce4dd-0ed3-4a5d-bc09-9629c0118fb8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:27:26.144098 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:27:26.174118 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-497ce4dd-0ed3-4a5d-bc09-9629c0118fb8 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:27:26.175349 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-497ce4dd-0ed3-4a5d-bc09-9629c0118fb8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:27:26.279108 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 722/1442] 127.0.0.1 () {58 vars in 1014 bytes} [Mon Aug 29 21:27:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:27:26.571197 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-541e2ff6-1074-4bbf-b57e-68c5d9093e5b None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:27:26.572742 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-541e2ff6-1074-4bbf-b57e-68c5d9093e5b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:27:26.679607 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-541e2ff6-1074-4bbf-b57e-68c5d9093e5b None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:27:26.712423 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:27:26.755144 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba250f1d-f18b-4895-9a6d-038346a783ae service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:27:26.755761 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba250f1d-f18b-4895-9a6d-038346a783ae service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:27:26.756043 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba250f1d-f18b-4895-9a6d-038346a783ae service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:27:26.756320 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba250f1d-f18b-4895-9a6d-038346a783ae service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:27:26.757989 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-ba250f1d-f18b-4895-9a6d-038346a783ae service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:27:26.889968 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba250f1d-f18b-4895-9a6d-038346a783ae service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:27:26.891399 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba250f1d-f18b-4895-9a6d-038346a783ae service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:27:26.910016 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 721/1443] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:27:26 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 339 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:27:29.022680 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db865d98-7d0a-4649-87e3-ca0040e54292 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:27:29.023208 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db865d98-7d0a-4649-87e3-ca0040e54292 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:27:29.023208 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db865d98-7d0a-4649-87e3-ca0040e54292 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:27:29.024576 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 723/1444] 127.0.0.1 () {54 vars in 917 bytes} [Mon Aug 29 21:27:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:27:29.029973 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9faa32b-117f-435f-8d2f-99018a62d672 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:27:29.030394 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9faa32b-117f-435f-8d2f-99018a62d672 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:27:29.030509 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9faa32b-117f-435f-8d2f-99018a62d672 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:27:29.106831 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-b9faa32b-117f-435f-8d2f-99018a62d672 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:27:29.108065 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-b9faa32b-117f-435f-8d2f-99018a62d672 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:27:29.121474 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:27:29.206935 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 722/1445] 127.0.0.1 () {58 vars in 1014 bytes} [Mon Aug 29 21:27:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:27:29.495320 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db865d98-7d0a-4649-87e3-ca0040e54292 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:27:29.496858 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-db865d98-7d0a-4649-87e3-ca0040e54292 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:27:29.603178 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db865d98-7d0a-4649-87e3-ca0040e54292 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:27:29.616314 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:27:29.679571 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6404e96a-d844-46f0-a648-5d7508bee3dc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:27:29.680249 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6404e96a-d844-46f0-a648-5d7508bee3dc service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:27:29.680524 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6404e96a-d844-46f0-a648-5d7508bee3dc service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:27:29.680802 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6404e96a-d844-46f0-a648-5d7508bee3dc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:27:29.682422 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-6404e96a-d844-46f0-a648-5d7508bee3dc service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:27:29.790625 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6404e96a-d844-46f0-a648-5d7508bee3dc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:27:29.792060 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6404e96a-d844-46f0-a648-5d7508bee3dc service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:27:29.810213 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 724/1446] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:27:29 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 315 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:27:32.054393 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13e94533-183b-4f8b-b1e2-2097bf547ede None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:27:32.054932 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13e94533-183b-4f8b-b1e2-2097bf547ede None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:27:32.054932 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13e94533-183b-4f8b-b1e2-2097bf547ede None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:27:32.056267 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 723/1447] 127.0.0.1 () {54 vars in 917 bytes} [Mon Aug 29 21:27:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:27:32.061669 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f922f35-9957-4fdd-a2ce-7211869755ef None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:27:32.062194 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f922f35-9957-4fdd-a2ce-7211869755ef None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:27:32.062290 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f922f35-9957-4fdd-a2ce-7211869755ef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:27:32.113209 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:27:32.132521 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-1f922f35-9957-4fdd-a2ce-7211869755ef None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:27:32.133805 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-1f922f35-9957-4fdd-a2ce-7211869755ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:27:32.247287 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 725/1448] 127.0.0.1 () {58 vars in 1014 bytes} [Mon Aug 29 21:27:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:27:32.554770 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13e94533-183b-4f8b-b1e2-2097bf547ede None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:27:32.556324 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-13e94533-183b-4f8b-b1e2-2097bf547ede None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:27:32.667744 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13e94533-183b-4f8b-b1e2-2097bf547ede None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:27:32.709800 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:27:32.745581 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-984f24c6-9f86-4a22-83b4-3098921f87c2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:27:32.746231 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-984f24c6-9f86-4a22-83b4-3098921f87c2 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:27:32.746509 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-984f24c6-9f86-4a22-83b4-3098921f87c2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:27:32.746781 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-984f24c6-9f86-4a22-83b4-3098921f87c2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:27:32.748412 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-984f24c6-9f86-4a22-83b4-3098921f87c2 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:27:32.871284 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-984f24c6-9f86-4a22-83b4-3098921f87c2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:27:32.872737 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-984f24c6-9f86-4a22-83b4-3098921f87c2 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:27:32.893734 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 724/1449] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:27:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 340 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 21:27:34.965846 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68e3458b-0a9e-4420-a4c7-10d701c19b63 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:27:34.966388 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68e3458b-0a9e-4420-a4c7-10d701c19b63 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:27:34.966388 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68e3458b-0a9e-4420-a4c7-10d701c19b63 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:27:34.967714 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 726/1450] 127.0.0.1 () {54 vars in 917 bytes} [Mon Aug 29 21:27:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:27:34.973561 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39a09870-cf91-4418-a598-a1e5728b8eb7 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:27:34.974085 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39a09870-cf91-4418-a598-a1e5728b8eb7 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:27:34.974186 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39a09870-cf91-4418-a598-a1e5728b8eb7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:27:35.043398 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-39a09870-cf91-4418-a598-a1e5728b8eb7 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:27:35.044647 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-39a09870-cf91-4418-a598-a1e5728b8eb7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:27:35.066456 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:27:35.145540 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 725/1451] 127.0.0.1 () {58 vars in 1014 bytes} [Mon Aug 29 21:27:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:27:35.440615 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68e3458b-0a9e-4420-a4c7-10d701c19b63 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:27:35.442287 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-68e3458b-0a9e-4420-a4c7-10d701c19b63 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:27:35.550380 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68e3458b-0a9e-4420-a4c7-10d701c19b63 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:27:35.570699 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:27:35.631701 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d32fb52-eb83-46bb-9837-be7a212ad0ea service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:27:35.632316 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d32fb52-eb83-46bb-9837-be7a212ad0ea service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:27:35.632605 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d32fb52-eb83-46bb-9837-be7a212ad0ea service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:27:35.632880 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d32fb52-eb83-46bb-9837-be7a212ad0ea service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:27:35.634658 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-5d32fb52-eb83-46bb-9837-be7a212ad0ea service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:27:35.750078 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d32fb52-eb83-46bb-9837-be7a212ad0ea service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:27:35.751432 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d32fb52-eb83-46bb-9837-be7a212ad0ea service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:27:35.770489 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 727/1452] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:27:35 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 332 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:27:37.816092 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6f33923-ba81-414c-86fa-671c2a72d628 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:27:37.816952 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6f33923-ba81-414c-86fa-671c2a72d628 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:27:37.816952 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6f33923-ba81-414c-86fa-671c2a72d628 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:27:37.818038 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 726/1453] 127.0.0.1 () {54 vars in 917 bytes} [Mon Aug 29 21:27:37 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:27:37.824188 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab030741-1084-4f8c-ac81-82272112d0c8 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:27:37.824726 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab030741-1084-4f8c-ac81-82272112d0c8 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:27:37.824726 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab030741-1084-4f8c-ac81-82272112d0c8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:27:37.898402 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:27:37.903813 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-ab030741-1084-4f8c-ac81-82272112d0c8 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:27:37.906187 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-ab030741-1084-4f8c-ac81-82272112d0c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:27:38.041976 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 728/1454] 127.0.0.1 () {58 vars in 1014 bytes} [Mon Aug 29 21:27:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 219 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:27:38.337105 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6f33923-ba81-414c-86fa-671c2a72d628 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:27:38.338906 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-e6f33923-ba81-414c-86fa-671c2a72d628 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:27:38.449501 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6f33923-ba81-414c-86fa-671c2a72d628 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:27:38.504932 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:27:38.530649 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd4e5e90-3dd3-46dd-a5b2-a89f3f3b9bed service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:27:38.531265 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd4e5e90-3dd3-46dd-a5b2-a89f3f3b9bed service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:27:38.531545 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd4e5e90-3dd3-46dd-a5b2-a89f3f3b9bed service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:27:38.531817 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd4e5e90-3dd3-46dd-a5b2-a89f3f3b9bed service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:27:38.534012 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-bd4e5e90-3dd3-46dd-a5b2-a89f3f3b9bed service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:27:38.667317 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd4e5e90-3dd3-46dd-a5b2-a89f3f3b9bed service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:27:38.668767 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd4e5e90-3dd3-46dd-a5b2-a89f3f3b9bed service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:27:38.692303 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 727/1455] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:27:38 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 356 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:27:40.748937 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2dad0e6-f34e-4f8b-b2a4-8b93b01a964b None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:27:40.749500 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2dad0e6-f34e-4f8b-b2a4-8b93b01a964b None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:27:40.749500 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2dad0e6-f34e-4f8b-b2a4-8b93b01a964b None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:27:40.750884 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 729/1456] 127.0.0.1 () {54 vars in 917 bytes} [Mon Aug 29 21:27:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:27:40.756216 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-088f6cba-1381-4a84-be9a-455c779ca7f4 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:27:40.756693 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-088f6cba-1381-4a84-be9a-455c779ca7f4 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:27:40.756793 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-088f6cba-1381-4a84-be9a-455c779ca7f4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:27:40.825020 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-088f6cba-1381-4a84-be9a-455c779ca7f4 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:27:40.826287 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-088f6cba-1381-4a84-be9a-455c779ca7f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:27:40.857815 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:27:40.927216 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 728/1457] 127.0.0.1 () {58 vars in 1014 bytes} [Mon Aug 29 21:27:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:27:41.211432 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2dad0e6-f34e-4f8b-b2a4-8b93b01a964b None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:27:41.213220 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-d2dad0e6-f34e-4f8b-b2a4-8b93b01a964b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:27:41.317837 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2dad0e6-f34e-4f8b-b2a4-8b93b01a964b None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:27:41.347838 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:27:41.395252 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26d59b86-90dc-479e-aebf-8ddfe4b4647d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:27:41.395869 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26d59b86-90dc-479e-aebf-8ddfe4b4647d service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:27:41.396153 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26d59b86-90dc-479e-aebf-8ddfe4b4647d service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:27:41.396433 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26d59b86-90dc-479e-aebf-8ddfe4b4647d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:27:41.398086 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-26d59b86-90dc-479e-aebf-8ddfe4b4647d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:27:41.507558 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26d59b86-90dc-479e-aebf-8ddfe4b4647d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:27:41.508916 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26d59b86-90dc-479e-aebf-8ddfe4b4647d service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:27:41.527505 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 730/1458] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:27:41 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 317 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:27:43.540659 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ecac3f1-0f7f-4e53-8fd4-74f67b89807f None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:27:43.541511 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ecac3f1-0f7f-4e53-8fd4-74f67b89807f None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:27:43.541511 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ecac3f1-0f7f-4e53-8fd4-74f67b89807f None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:27:43.542809 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 729/1459] 127.0.0.1 () {54 vars in 917 bytes} [Mon Aug 29 21:27:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:27:43.548972 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da988904-3b93-47e8-b100-00152a09f4fe None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:27:43.549565 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da988904-3b93-47e8-b100-00152a09f4fe None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:27:43.550160 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da988904-3b93-47e8-b100-00152a09f4fe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:27:43.620911 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-da988904-3b93-47e8-b100-00152a09f4fe None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:27:43.622207 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-da988904-3b93-47e8-b100-00152a09f4fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:27:43.631486 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:27:43.727979 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 731/1460] 127.0.0.1 () {58 vars in 1014 bytes} [Mon Aug 29 21:27:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 21:27:44.013814 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ecac3f1-0f7f-4e53-8fd4-74f67b89807f None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:27:44.015403 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-4ecac3f1-0f7f-4e53-8fd4-74f67b89807f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:27:44.123126 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ecac3f1-0f7f-4e53-8fd4-74f67b89807f None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:27:44.182613 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:27:44.213724 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52291a0b-f2a2-4342-834b-b3172ba325f4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:27:44.214377 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52291a0b-f2a2-4342-834b-b3172ba325f4 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:27:44.214660 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52291a0b-f2a2-4342-834b-b3172ba325f4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:27:44.214938 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52291a0b-f2a2-4342-834b-b3172ba325f4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:27:44.216585 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-52291a0b-f2a2-4342-834b-b3172ba325f4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:27:44.333575 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52291a0b-f2a2-4342-834b-b3172ba325f4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:27:44.335058 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52291a0b-f2a2-4342-834b-b3172ba325f4 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:27:44.355036 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 730/1461] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:27:44 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 342 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:27:46.382086 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48c2fbb2-dbec-485d-a0ae-8f144e85d8a2 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:27:46.382631 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48c2fbb2-dbec-485d-a0ae-8f144e85d8a2 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:27:46.382631 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48c2fbb2-dbec-485d-a0ae-8f144e85d8a2 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:27:46.384000 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 732/1462] 127.0.0.1 () {54 vars in 917 bytes} [Mon Aug 29 21:27:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:27:46.389982 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25b9230b-b8e2-4a8f-9718-6e25f64f37c2 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:27:46.390449 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25b9230b-b8e2-4a8f-9718-6e25f64f37c2 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:27:46.390570 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25b9230b-b8e2-4a8f-9718-6e25f64f37c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:27:46.464195 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-25b9230b-b8e2-4a8f-9718-6e25f64f37c2 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:27:46.465463 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-25b9230b-b8e2-4a8f-9718-6e25f64f37c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:27:46.510604 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:27:46.578210 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 731/1463] 127.0.0.1 () {58 vars in 1014 bytes} [Mon Aug 29 21:27:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:27:46.871134 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48c2fbb2-dbec-485d-a0ae-8f144e85d8a2 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:27:46.873273 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-48c2fbb2-dbec-485d-a0ae-8f144e85d8a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:27:47.016123 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48c2fbb2-dbec-485d-a0ae-8f144e85d8a2 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:27:47.056134 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:27:47.097221 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-538645c2-3878-4f1f-bf8d-a26c9da49f58 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:27:47.097863 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-538645c2-3878-4f1f-bf8d-a26c9da49f58 service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:27:47.098149 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-538645c2-3878-4f1f-bf8d-a26c9da49f58 service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:27:47.098414 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-538645c2-3878-4f1f-bf8d-a26c9da49f58 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:27:47.100056 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-538645c2-3878-4f1f-bf8d-a26c9da49f58 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:27:47.222320 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-538645c2-3878-4f1f-bf8d-a26c9da49f58 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:27:47.223677 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-538645c2-3878-4f1f-bf8d-a26c9da49f58 service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:27:47.244455 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 733/1464] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:27:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 374 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:27:49.456799 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9921bfc-2985-4ea0-b488-f19893de41ca None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:27:49.457403 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9921bfc-2985-4ea0-b488-f19893de41ca None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:27:49.457755 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9921bfc-2985-4ea0-b488-f19893de41ca None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:27:49.459316 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 732/1465] 127.0.0.1 () {54 vars in 917 bytes} [Mon Aug 29 21:27:49 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:27:49.466342 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1069779b-e6e5-40ab-87cd-822d682eb0d3 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:27:49.467079 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1069779b-e6e5-40ab-87cd-822d682eb0d3 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:27:49.467079 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1069779b-e6e5-40ab-87cd-822d682eb0d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:27:49.548269 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-1069779b-e6e5-40ab-87cd-822d682eb0d3 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:27:49.550483 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-1069779b-e6e5-40ab-87cd-822d682eb0d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:27:49.571956 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:27:49.668437 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 734/1466] 127.0.0.1 () {58 vars in 1014 bytes} [Mon Aug 29 21:27:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 204 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:27:49.971284 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9921bfc-2985-4ea0-b488-f19893de41ca None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:27:49.972991 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-d9921bfc-2985-4ea0-b488-f19893de41ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:27:50.158561 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9921bfc-2985-4ea0-b488-f19893de41ca None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:27:50.254309 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:27:50.263366 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cc551bb-9f08-4686-9141-04a95700c70e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:27:50.264061 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc551bb-9f08-4686-9141-04a95700c70e service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:27:50.264389 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc551bb-9f08-4686-9141-04a95700c70e service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:27:50.264764 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc551bb-9f08-4686-9141-04a95700c70e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:27:50.266753 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-3cc551bb-9f08-4686-9141-04a95700c70e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:27:50.419854 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cc551bb-9f08-4686-9141-04a95700c70e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:27:50.421463 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cc551bb-9f08-4686-9141-04a95700c70e service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:27:50.444013 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 733/1467] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:27:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 474 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 21:27:52.686619 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26b1624a-419f-4abe-b3d1-e8060202f0a0 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:27:52.687196 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26b1624a-419f-4abe-b3d1-e8060202f0a0 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:27:52.687467 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26b1624a-419f-4abe-b3d1-e8060202f0a0 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:27:52.691160 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 735/1468] 127.0.0.1 () {54 vars in 917 bytes} [Mon Aug 29 21:27:52 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:27:52.700093 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0804f03-8419-4eeb-a240-52e4319e2719 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:27:52.700939 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0804f03-8419-4eeb-a240-52e4319e2719 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:27:52.701123 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0804f03-8419-4eeb-a240-52e4319e2719 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:27:52.812249 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-b0804f03-8419-4eeb-a240-52e4319e2719 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:27:52.814873 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-b0804f03-8419-4eeb-a240-52e4319e2719 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:27:52.908966 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:27:52.972305 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 734/1469] 127.0.0.1 () {58 vars in 1014 bytes} [Mon Aug 29 21:27:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 273 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 21:27:53.261207 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26b1624a-419f-4abe-b3d1-e8060202f0a0 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:27:53.262857 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-26b1624a-419f-4abe-b3d1-e8060202f0a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:27:53.398850 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26b1624a-419f-4abe-b3d1-e8060202f0a0 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:27:53.455864 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:27:53.493927 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbcff2d1-9ed0-4b70-95be-3fe4e109feee service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:27:53.494565 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbcff2d1-9ed0-4b70-95be-3fe4e109feee service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:27:53.494852 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbcff2d1-9ed0-4b70-95be-3fe4e109feee service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:27:53.495125 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbcff2d1-9ed0-4b70-95be-3fe4e109feee service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:27:53.496831 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-dbcff2d1-9ed0-4b70-95be-3fe4e109feee service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:27:53.652929 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbcff2d1-9ed0-4b70-95be-3fe4e109feee service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:27:53.655805 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbcff2d1-9ed0-4b70-95be-3fe4e109feee service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:27:53.688522 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 736/1470] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:27:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 428 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:28:02.970219 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=22 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:28:03.685753 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=66 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:28:14.753129 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b62402e2-712e-433a-a429-77cdd869ca5d None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:28:14.753707 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b62402e2-712e-433a-a429-77cdd869ca5d None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:28:14.753707 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b62402e2-712e-433a-a429-77cdd869ca5d None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:28:14.755027 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 735/1471] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:28:14 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:28:14.760476 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42964840-f548-4651-970a-4e7a04436057 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:28:14.760922 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42964840-f548-4651-970a-4e7a04436057 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:28:14.761040 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42964840-f548-4651-970a-4e7a04436057 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:28:14.827150 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-42964840-f548-4651-970a-4e7a04436057 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:28:14.828348 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-42964840-f548-4651-970a-4e7a04436057 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:28:14.929565 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 737/1472] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:28:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:28:15.241550 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74b31391-8ae1-49dd-954c-0d5a202efe9b None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:28:15.241874 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74b31391-8ae1-49dd-954c-0d5a202efe9b None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:28:15.242383 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74b31391-8ae1-49dd-954c-0d5a202efe9b None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:28:15.243509 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 736/1473] 10.222.0.90 () {54 vars in 967 bytes} [Mon Aug 29 21:28:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:28:15.248338 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5387ee0-018c-428e-b765-306557b324b9 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:28:15.248630 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5387ee0-018c-428e-b765-306557b324b9 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:28:15.248912 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5387ee0-018c-428e-b765-306557b324b9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:28:15.325690 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-f5387ee0-018c-428e-b765-306557b324b9 None None] MFA Rules not processed for user `c63a4eaf48ad43529f0aa328f2ebfd29`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:28:15.326927 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-f5387ee0-018c-428e-b765-306557b324b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:28:15.349588 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:28:15.412666 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 738/1474] 10.222.0.90 () {58 vars in 1064 bytes} [Mon Aug 29 21:28:15 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:28:15.420119 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74b31391-8ae1-49dd-954c-0d5a202efe9b None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:28:15.421672 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-74b31391-8ae1-49dd-954c-0d5a202efe9b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:28:15.524916 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74b31391-8ae1-49dd-954c-0d5a202efe9b None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:28:15.599673 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2f00460-a5d5-4c63-b0e1-42192387dfc2 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c63a4eaf48ad43529f0aa328f2ebfd29', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:28:15.600292 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2f00460-a5d5-4c63-b0e1-42192387dfc2 service glance] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:28:15.600579 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2f00460-a5d5-4c63-b0e1-42192387dfc2 service glance] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:28:15.600856 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2f00460-a5d5-4c63-b0e1-42192387dfc2 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:28:15.602520 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-f2f00460-a5d5-4c63-b0e1-42192387dfc2 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:28:15.655939 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:28:15.711979 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2f00460-a5d5-4c63-b0e1-42192387dfc2 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:28:15.713672 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2f00460-a5d5-4c63-b0e1-42192387dfc2 service glance] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:28:15.732558 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 737/1475] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:28:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 313 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:28:17.781035 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32572001-018a-4e8b-acd6-714e81b97fa9 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:28:17.781545 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32572001-018a-4e8b-acd6-714e81b97fa9 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:28:17.781671 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32572001-018a-4e8b-acd6-714e81b97fa9 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:28:17.782952 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 739/1476] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:28:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:28:17.788281 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef380aea-6475-4907-bd36-b9182e8bccf2 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:28:17.788737 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef380aea-6475-4907-bd36-b9182e8bccf2 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:28:17.788845 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef380aea-6475-4907-bd36-b9182e8bccf2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:28:17.856643 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-ef380aea-6475-4907-bd36-b9182e8bccf2 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:28:17.857900 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-ef380aea-6475-4907-bd36-b9182e8bccf2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:28:17.956822 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 738/1477] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:28:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:28:18.234859 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32572001-018a-4e8b-acd6-714e81b97fa9 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:28:18.237553 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-32572001-018a-4e8b-acd6-714e81b97fa9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:28:18.369394 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32572001-018a-4e8b-acd6-714e81b97fa9 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:28:18.420464 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:28:18.467655 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b22886ce-3aa0-4851-a6a6-ef24bf5f3091 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:28:18.468272 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b22886ce-3aa0-4851-a6a6-ef24bf5f3091 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:28:18.468551 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b22886ce-3aa0-4851-a6a6-ef24bf5f3091 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:28:18.468827 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b22886ce-3aa0-4851-a6a6-ef24bf5f3091 admin admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:28:18.469472 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b22886ce-3aa0-4851-a6a6-ef24bf5f3091 admin admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:28:18.471870 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: WARNING py.warnings [None req-b22886ce-3aa0-4851-a6a6-ef24bf5f3091 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_projects": "(role:reader and system_scope:all) or (role:reader and domain_id:%(target.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 29 21:28:18.471870 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: warnings.warn(msg) Aug 29 21:28:18.471870 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: Aug 29 21:28:18.473017 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b22886ce-3aa0-4851-a6a6-ef24bf5f3091 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:28:18.494561 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 740/1478] 10.222.0.90 () {56 vars in 1108 bytes} [Mon Aug 29 21:28:18 2022] GET /identity/v3/projects => generated 2553 bytes in 260 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:28:20.456432 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aceaf02-fb0d-4ba8-bef4-cb286ea7137b None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:28:20.457109 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aceaf02-fb0d-4ba8-bef4-cb286ea7137b None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:28:20.457109 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aceaf02-fb0d-4ba8-bef4-cb286ea7137b None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:28:20.458283 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 739/1479] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:28:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:28:20.463993 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98307fdf-55b7-43bd-9cc3-df333e0969ee None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:28:20.464423 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98307fdf-55b7-43bd-9cc3-df333e0969ee None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:28:20.464528 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98307fdf-55b7-43bd-9cc3-df333e0969ee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:28:20.531836 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-98307fdf-55b7-43bd-9cc3-df333e0969ee None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:28:20.533604 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-98307fdf-55b7-43bd-9cc3-df333e0969ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:28:20.631590 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 741/1480] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:28:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 21:28:20.931306 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aceaf02-fb0d-4ba8-bef4-cb286ea7137b None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:28:20.932829 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-5aceaf02-fb0d-4ba8-bef4-cb286ea7137b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:28:20.943450 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:28:21.080534 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aceaf02-fb0d-4ba8-bef4-cb286ea7137b None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:28:21.164558 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:28:21.199443 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee8ec958-d10c-4cb4-a88b-78d2432a0e4c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a35b5b7dea7b421da056f063d2436ffb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:28:21.200127 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee8ec958-d10c-4cb4-a88b-78d2432a0e4c service nova] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:28:21.200567 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee8ec958-d10c-4cb4-a88b-78d2432a0e4c service nova] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:28:21.200909 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee8ec958-d10c-4cb4-a88b-78d2432a0e4c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:28:21.202734 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-ee8ec958-d10c-4cb4-a88b-78d2432a0e4c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:28:21.292463 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee8ec958-d10c-4cb4-a88b-78d2432a0e4c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:28:21.294145 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee8ec958-d10c-4cb4-a88b-78d2432a0e4c service nova] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:28:21.312879 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 740/1481] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:28:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 382 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:28:23.433004 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b310e87e-7b4d-426e-b0f1-1d9993233699 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:28:23.433579 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b310e87e-7b4d-426e-b0f1-1d9993233699 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:28:23.433579 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b310e87e-7b4d-426e-b0f1-1d9993233699 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:28:23.434937 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 742/1482] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:28:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:28:23.441111 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a9a6c14-7e36-438c-a000-26340684351e None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:28:23.441569 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a9a6c14-7e36-438c-a000-26340684351e None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:28:23.441676 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a9a6c14-7e36-438c-a000-26340684351e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:28:23.514137 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-3a9a6c14-7e36-438c-a000-26340684351e None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:28:23.515363 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-3a9a6c14-7e36-438c-a000-26340684351e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:28:23.563156 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:28:23.614489 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 741/1483] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:28:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:28:23.905192 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b310e87e-7b4d-426e-b0f1-1d9993233699 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:28:23.906799 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-b310e87e-7b4d-426e-b0f1-1d9993233699 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:28:23.930708 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:28:24.010705 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b310e87e-7b4d-426e-b0f1-1d9993233699 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:28:24.083584 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9326e1da-9531-4155-ae6c-69690abf7eb4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:28:24.084258 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9326e1da-9531-4155-ae6c-69690abf7eb4 service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:28:24.084600 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9326e1da-9531-4155-ae6c-69690abf7eb4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:28:24.084929 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9326e1da-9531-4155-ae6c-69690abf7eb4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:28:24.086717 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-9326e1da-9531-4155-ae6c-69690abf7eb4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:28:24.164173 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:28:24.199512 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9326e1da-9531-4155-ae6c-69690abf7eb4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:28:24.200948 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9326e1da-9531-4155-ae6c-69690abf7eb4 service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:28:24.220030 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 743/1484] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:28:23 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 315 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:28:26.226567 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a132927d-e942-4d5a-950d-ae8623765dbc None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:28:26.227133 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a132927d-e942-4d5a-950d-ae8623765dbc None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:28:26.227133 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a132927d-e942-4d5a-950d-ae8623765dbc None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:28:26.228462 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 742/1485] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:28:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:28:26.233688 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da076a47-0d11-44b2-966a-6a14c791cffb None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:28:26.234119 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da076a47-0d11-44b2-966a-6a14c791cffb None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:28:26.234284 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da076a47-0d11-44b2-966a-6a14c791cffb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:28:26.301303 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-da076a47-0d11-44b2-966a-6a14c791cffb None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:28:26.302581 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-da076a47-0d11-44b2-966a-6a14c791cffb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:28:26.401811 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 744/1486] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:28:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:28:26.700701 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a132927d-e942-4d5a-950d-ae8623765dbc None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:28:26.702324 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-a132927d-e942-4d5a-950d-ae8623765dbc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:28:26.806886 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a132927d-e942-4d5a-950d-ae8623765dbc None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:28:26.882400 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:28:26.884301 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb4ddaad-0c3b-4f85-a2ca-f6e633d1e368 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:28:26.884984 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb4ddaad-0c3b-4f85-a2ca-f6e633d1e368 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:28:26.885276 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb4ddaad-0c3b-4f85-a2ca-f6e633d1e368 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:28:26.885737 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb4ddaad-0c3b-4f85-a2ca-f6e633d1e368 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:28:26.887731 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-bb4ddaad-0c3b-4f85-a2ca-f6e633d1e368 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:28:27.118509 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb4ddaad-0c3b-4f85-a2ca-f6e633d1e368 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:28:27.119929 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb4ddaad-0c3b-4f85-a2ca-f6e633d1e368 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:28:27.137916 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 743/1487] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:28:26 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 438 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:28:29.289975 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb393241-202c-4528-886a-aedc1c07200d None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:28:29.290588 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb393241-202c-4528-886a-aedc1c07200d None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:28:29.290588 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb393241-202c-4528-886a-aedc1c07200d None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:28:29.291879 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 745/1488] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:28:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:28:29.297206 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ee1538d-107e-42a3-bb3e-3dbcd98641af None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:28:29.297615 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ee1538d-107e-42a3-bb3e-3dbcd98641af None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:28:29.297758 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ee1538d-107e-42a3-bb3e-3dbcd98641af None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:28:29.366773 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-8ee1538d-107e-42a3-bb3e-3dbcd98641af None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:28:29.368008 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-8ee1538d-107e-42a3-bb3e-3dbcd98641af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:28:29.427842 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:28:29.474897 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 744/1489] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:28:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:28:29.775853 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb393241-202c-4528-886a-aedc1c07200d None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:28:29.777168 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-bb393241-202c-4528-886a-aedc1c07200d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:28:29.819775 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:28:29.891011 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb393241-202c-4528-886a-aedc1c07200d None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:28:29.967626 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02517347-27c1-46c3-95c1-e0ebbd75c8b3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:28:29.968229 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02517347-27c1-46c3-95c1-e0ebbd75c8b3 service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:28:29.968507 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02517347-27c1-46c3-95c1-e0ebbd75c8b3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:28:29.968778 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02517347-27c1-46c3-95c1-e0ebbd75c8b3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:28:29.970461 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-02517347-27c1-46c3-95c1-e0ebbd75c8b3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:28:30.060102 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:28:30.091805 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02517347-27c1-46c3-95c1-e0ebbd75c8b3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:28:30.093758 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02517347-27c1-46c3-95c1-e0ebbd75c8b3 service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:28:30.114472 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 746/1490] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:28:29 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 340 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:28:32.264692 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb677ed-2463-4044-9b55-5215104bdc1a None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:28:32.265210 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb677ed-2463-4044-9b55-5215104bdc1a None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:28:32.265210 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb677ed-2463-4044-9b55-5215104bdc1a None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:28:32.266515 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 745/1491] 10.222.0.90 () {54 vars in 927 bytes} [Mon Aug 29 21:28:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:28:32.272438 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-940e6cad-db73-4392-8f79-37f54e76eea8 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:28:32.272898 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-940e6cad-db73-4392-8f79-37f54e76eea8 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:28:32.273007 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-940e6cad-db73-4392-8f79-37f54e76eea8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:28:32.342677 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-940e6cad-db73-4392-8f79-37f54e76eea8 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:28:32.343893 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-940e6cad-db73-4392-8f79-37f54e76eea8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:28:32.454169 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 747/1492] 10.222.0.90 () {58 vars in 1024 bytes} [Mon Aug 29 21:28:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:28:32.755874 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feb677ed-2463-4044-9b55-5215104bdc1a None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:28:32.757463 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-feb677ed-2463-4044-9b55-5215104bdc1a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:28:32.879721 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feb677ed-2463-4044-9b55-5215104bdc1a None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:28:32.964922 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9baa0d1-5ba2-4ef4-959a-5b221922032f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:28:32.965441 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9baa0d1-5ba2-4ef4-959a-5b221922032f service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:28:32.965716 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9baa0d1-5ba2-4ef4-959a-5b221922032f service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:28:32.966009 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9baa0d1-5ba2-4ef4-959a-5b221922032f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:28:32.967609 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-f9baa0d1-5ba2-4ef4-959a-5b221922032f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:28:32.976821 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:28:33.094688 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9baa0d1-5ba2-4ef4-959a-5b221922032f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:28:33.096393 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9baa0d1-5ba2-4ef4-959a-5b221922032f service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:28:33.117570 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 746/1493] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:28:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 362 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:28:42.452235 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=82 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:28:43.115294 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=50 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:29:51.681709 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f61ff088-ce4f-4e3b-bd9b-6c7551484a81 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:29:51.682432 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f61ff088-ce4f-4e3b-bd9b-6c7551484a81 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:29:51.682432 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f61ff088-ce4f-4e3b-bd9b-6c7551484a81 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:29:51.765016 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-f61ff088-ce4f-4e3b-bd9b-6c7551484a81 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:29:51.766832 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-f61ff088-ce4f-4e3b-bd9b-6c7551484a81 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:29:52.000936 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 748/1494] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:29:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 320 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:29:52.009186 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be194898-96aa-4edf-952a-01e51ddc87b1 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:29:52.009748 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be194898-96aa-4edf-952a-01e51ddc87b1 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:29:52.009748 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be194898-96aa-4edf-952a-01e51ddc87b1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:29:52.088500 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-be194898-96aa-4edf-952a-01e51ddc87b1 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:29:52.090019 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-be194898-96aa-4edf-952a-01e51ddc87b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:29:52.190582 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 747/1495] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:29:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:29:52.195477 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f61ff088-ce4f-4e3b-bd9b-6c7551484a81 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:29:52.196696 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-f61ff088-ce4f-4e3b-bd9b-6c7551484a81 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:29:52.209292 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f61ff088-ce4f-4e3b-bd9b-6c7551484a81 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:29:52.274170 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:29:52.309895 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b68253c0-661f-4710-92fd-4d91d830f302 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:29:52.310511 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b68253c0-661f-4710-92fd-4d91d830f302 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:29:52.310791 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b68253c0-661f-4710-92fd-4d91d830f302 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:29:52.311076 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b68253c0-661f-4710-92fd-4d91d830f302 admin admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:29:52.311722 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b68253c0-661f-4710-92fd-4d91d830f302 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:29:52.312026 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b68253c0-661f-4710-92fd-4d91d830f302 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:29:52.313080 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: WARNING py.warnings [None req-b68253c0-661f-4710-92fd-4d91d830f302 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Aug 29 21:29:52.313080 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: warnings.warn(msg) Aug 29 21:29:52.313080 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: Aug 29 21:29:52.314170 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b68253c0-661f-4710-92fd-4d91d830f302 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:29:52.326430 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 749/1496] 10.222.0.90 () {58 vars in 1153 bytes} [Mon Aug 29 21:29:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 131 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:29:52.335570 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be194898-96aa-4edf-952a-01e51ddc87b1 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:29:52.345979 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be194898-96aa-4edf-952a-01e51ddc87b1 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:29:52.408952 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:29:52.443711 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e523db8-6672-4c10-87b4-1cc4bcaf72c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:29:52.444325 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e523db8-6672-4c10-87b4-1cc4bcaf72c3 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:29:52.444602 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e523db8-6672-4c10-87b4-1cc4bcaf72c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:29:52.444867 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e523db8-6672-4c10-87b4-1cc4bcaf72c3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:29:52.445499 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e523db8-6672-4c10-87b4-1cc4bcaf72c3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:29:52.447393 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: WARNING py.warnings [None req-9e523db8-6672-4c10-87b4-1cc4bcaf72c3 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 29 21:29:52.447393 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: warnings.warn(msg) Aug 29 21:29:52.447393 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: Aug 29 21:29:52.448584 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e523db8-6672-4c10-87b4-1cc4bcaf72c3 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:29:52.508977 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 748/1497] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:29:52 2022] POST /identity/v3/projects => generated 379 bytes in 174 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:29:52.513204 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b68253c0-661f-4710-92fd-4d91d830f302 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:29:52.514763 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-b68253c0-661f-4710-92fd-4d91d830f302 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:29:52.548979 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b68253c0-661f-4710-92fd-4d91d830f302 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:29:52.645868 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-175126d4-424d-4ce1-abbf-675ca0c470e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:29:52.646483 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-175126d4-424d-4ce1-abbf-675ca0c470e8 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:29:52.646791 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-175126d4-424d-4ce1-abbf-675ca0c470e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:29:52.647038 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-175126d4-424d-4ce1-abbf-675ca0c470e8 admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:29:52.647679 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-175126d4-424d-4ce1-abbf-675ca0c470e8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:29:52.648691 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: WARNING py.warnings [None req-175126d4-424d-4ce1-abbf-675ca0c470e8 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 29 21:29:52.648691 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: warnings.warn(msg) Aug 29 21:29:52.648691 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: Aug 29 21:29:52.649906 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-175126d4-424d-4ce1-abbf-675ca0c470e8 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:29:52.695115 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 750/1498] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:29:52 2022] POST /identity/v3/users => generated 290 bytes in 182 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:29:52.699885 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e523db8-6672-4c10-87b4-1cc4bcaf72c3 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:29:52.701419 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-9e523db8-6672-4c10-87b4-1cc4bcaf72c3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:29:52.773637 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e523db8-6672-4c10-87b4-1cc4bcaf72c3 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:29:52.864315 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:29:52.879556 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-695c17e5-0114-4453-a144-995ba9f0b1b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:29:52.880168 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-695c17e5-0114-4453-a144-995ba9f0b1b9 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:29:52.880443 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-695c17e5-0114-4453-a144-995ba9f0b1b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:29:52.880710 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-695c17e5-0114-4453-a144-995ba9f0b1b9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:29:52.881548 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-695c17e5-0114-4453-a144-995ba9f0b1b9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:29:52.882712 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: WARNING py.warnings [None req-695c17e5-0114-4453-a144-995ba9f0b1b9 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Aug 29 21:29:52.882712 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: warnings.warn(msg) Aug 29 21:29:52.882712 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: Aug 29 21:29:52.883622 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-695c17e5-0114-4453-a144-995ba9f0b1b9 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:29:52.896811 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 749/1499] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:29:52 2022] GET /identity/v3/roles => generated 1420 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:29:52.901615 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-175126d4-424d-4ce1-abbf-675ca0c470e8 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:29:52.903162 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-175126d4-424d-4ce1-abbf-675ca0c470e8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:29:52.915543 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-175126d4-424d-4ce1-abbf-675ca0c470e8 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:29:52.944503 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:29:53.020800 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-036badbe-409a-49b4-b436-3f42fd4adfea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:29:53.021415 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-036badbe-409a-49b4-b436-3f42fd4adfea admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:29:53.021690 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-036badbe-409a-49b4-b436-3f42fd4adfea admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:29:53.021999 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-036badbe-409a-49b4-b436-3f42fd4adfea admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:29:53.022645 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-036badbe-409a-49b4-b436-3f42fd4adfea admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:29:53.023660 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: WARNING py.warnings [None req-036badbe-409a-49b4-b436-3f42fd4adfea admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Aug 29 21:29:53.023660 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: warnings.warn(msg) Aug 29 21:29:53.023660 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: Aug 29 21:29:53.024548 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-036badbe-409a-49b4-b436-3f42fd4adfea admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:29:53.040115 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 751/1500] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:29:52 2022] GET /identity/v3/roles => generated 1420 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:29:53.045304 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-695c17e5-0114-4453-a144-995ba9f0b1b9 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:29:53.047039 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-695c17e5-0114-4453-a144-995ba9f0b1b9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:29:53.085981 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-695c17e5-0114-4453-a144-995ba9f0b1b9 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:29:53.195608 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97eb909a-5d02-4d36-8383-22b538144020 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:29:53.196192 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97eb909a-5d02-4d36-8383-22b538144020 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:29:53.196507 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97eb909a-5d02-4d36-8383-22b538144020 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:29:53.196756 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97eb909a-5d02-4d36-8383-22b538144020 admin admin] PATH_INFO: `/v3/projects/5523be9f650a4e10bf46c772a891b892/users/ff3dd5b41f284026b516eb40f11707dd/roles/02e242d5520740d4b77a5f4008c7eb57` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:29:53.224138 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97eb909a-5d02-4d36-8383-22b538144020 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5523be9f650a4e10bf46c772a891b892, user_id=ff3dd5b41f284026b516eb40f11707dd, role_id=02e242d5520740d4b77a5f4008c7eb57)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:29:53.225166 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: WARNING py.warnings [None req-97eb909a-5d02-4d36-8383-22b538144020 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 29 21:29:53.225166 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: warnings.warn(msg) Aug 29 21:29:53.225166 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: Aug 29 21:29:53.227156 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97eb909a-5d02-4d36-8383-22b538144020 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:29:53.259358 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 750/1501] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:29:53 2022] PUT /identity/v3/projects/5523be9f650a4e10bf46c772a891b892/users/ff3dd5b41f284026b516eb40f11707dd/roles/02e242d5520740d4b77a5f4008c7eb57 => generated 0 bytes in 215 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:29:53.274922 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e757604f-41ba-45dd-9e79-12abb182f76d None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:29:53.274922 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e757604f-41ba-45dd-9e79-12abb182f76d None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:29:53.274922 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e757604f-41ba-45dd-9e79-12abb182f76d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:29:53.340719 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-e757604f-41ba-45dd-9e79-12abb182f76d None None] MFA Rules not processed for user `ff3dd5b41f284026b516eb40f11707dd`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:29:53.341988 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-e757604f-41ba-45dd-9e79-12abb182f76d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:29:53.421589 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 752/1502] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:29:53 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 21:29:53.435648 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-322b54ae-8f06-4c19-a397-b919329d9d60 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:29:53.435937 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-322b54ae-8f06-4c19-a397-b919329d9d60 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:29:53.436215 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-322b54ae-8f06-4c19-a397-b919329d9d60 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:29:53.437559 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 751/1503] 10.222.0.90 () {54 vars in 964 bytes} [Mon Aug 29 21:29:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:29:53.443221 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7259db54-1d93-4c5c-958e-383de1fe436b None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:29:53.443511 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7259db54-1d93-4c5c-958e-383de1fe436b None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:29:53.443791 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7259db54-1d93-4c5c-958e-383de1fe436b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:29:53.478660 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:29:53.533516 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-7259db54-1d93-4c5c-958e-383de1fe436b None None] MFA Rules not processed for user `0dc226f7ddcb49ecacb51ce57fdfc11b`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:29:53.534799 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-7259db54-1d93-4c5c-958e-383de1fe436b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:29:53.623106 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 753/1504] 10.222.0.90 () {58 vars in 1061 bytes} [Mon Aug 29 21:29:53 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:29:53.631174 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-322b54ae-8f06-4c19-a397-b919329d9d60 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:29:53.632766 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-322b54ae-8f06-4c19-a397-b919329d9d60 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:29:53.685735 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:29:53.746056 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-322b54ae-8f06-4c19-a397-b919329d9d60 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:29:53.820685 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c40cec96-b4b5-4fbe-a66a-4be3b97bf8c8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dc226f7ddcb49ecacb51ce57fdfc11b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:29:53.821279 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c40cec96-b4b5-4fbe-a66a-4be3b97bf8c8 service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:29:53.821556 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c40cec96-b4b5-4fbe-a66a-4be3b97bf8c8 service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:29:53.821857 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c40cec96-b4b5-4fbe-a66a-4be3b97bf8c8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:29:53.823474 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-c40cec96-b4b5-4fbe-a66a-4be3b97bf8c8 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:29:53.944016 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c40cec96-b4b5-4fbe-a66a-4be3b97bf8c8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:29:53.945483 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c40cec96-b4b5-4fbe-a66a-4be3b97bf8c8 service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:29:53.949055 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:29:53.964867 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 752/1505] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:29:53 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 334 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:29:54.010784 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0156a3ba-a0f0-43a9-bca8-a2c4fa69abf6 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:29:54.011233 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0156a3ba-a0f0-43a9-bca8-a2c4fa69abf6 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:29:54.011339 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0156a3ba-a0f0-43a9-bca8-a2c4fa69abf6 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:29:54.012594 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 754/1506] 10.222.0.90 () {52 vars in 830 bytes} [Mon Aug 29 21:29:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:29:54.017301 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c40cec96-b4b5-4fbe-a66a-4be3b97bf8c8 service cinder] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:29:54.018628 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-c40cec96-b4b5-4fbe-a66a-4be3b97bf8c8 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:29:54.125304 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c40cec96-b4b5-4fbe-a66a-4be3b97bf8c8 service cinder] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:29:54.223032 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-711c821b-4c65-438c-bfa8-10e42a94ad2b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:29:54.223657 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-711c821b-4c65-438c-bfa8-10e42a94ad2b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:29:54.223937 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-711c821b-4c65-438c-bfa8-10e42a94ad2b admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:29:54.224242 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-711c821b-4c65-438c-bfa8-10e42a94ad2b admin admin] PATH_INFO: `/v3/users/ff3dd5b41f284026b516eb40f11707dd` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:29:54.236684 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-711c821b-4c65-438c-bfa8-10e42a94ad2b admin admin] RBAC: Authorizing `identity:delete_user(user_id=ff3dd5b41f284026b516eb40f11707dd)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:29:54.237706 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: WARNING py.warnings [None req-711c821b-4c65-438c-bfa8-10e42a94ad2b admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 29 21:29:54.237706 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: warnings.warn(msg) Aug 29 21:29:54.237706 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: Aug 29 21:29:54.238960 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-711c821b-4c65-438c-bfa8-10e42a94ad2b admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:29:54.248946 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:29:54.308208 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-711c821b-4c65-438c-bfa8-10e42a94ad2b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ff3dd5b41f284026b516eb40f11707dd'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:29:54.332649 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-711c821b-4c65-438c-bfa8-10e42a94ad2b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ff3dd5b41f284026b516eb40f11707dd'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:29:54.333225 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-711c821b-4c65-438c-bfa8-10e42a94ad2b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ff3dd5b41f284026b516eb40f11707dd'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:29:54.333653 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-711c821b-4c65-438c-bfa8-10e42a94ad2b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ff3dd5b41f284026b516eb40f11707dd'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:29:54.348270 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-711c821b-4c65-438c-bfa8-10e42a94ad2b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ff3dd5b41f284026b516eb40f11707dd'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:29:54.364108 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 753/1507] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:29:54 2022] DELETE /identity/v3/users/ff3dd5b41f284026b516eb40f11707dd => generated 0 bytes in 348 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:29:54.368583 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a62f097-7054-484a-a714-073741f987f1 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:29:54.368869 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a62f097-7054-484a-a714-073741f987f1 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:29:54.369145 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a62f097-7054-484a-a714-073741f987f1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:29:54.427565 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-2a62f097-7054-484a-a714-073741f987f1 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:29:54.428847 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-2a62f097-7054-484a-a714-073741f987f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:29:54.514414 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:29:54.529421 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 755/1508] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:29:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:29:54.540772 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-711c821b-4c65-438c-bfa8-10e42a94ad2b admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:29:54.542061 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-711c821b-4c65-438c-bfa8-10e42a94ad2b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:29:54.656101 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-711c821b-4c65-438c-bfa8-10e42a94ad2b admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:29:54.722099 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:DELETE=10,keystone:SELECT=89,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:29:54.744591 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1276028-cc7e-497e-9b68-53e73e679e2e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:29:54.745503 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1276028-cc7e-497e-9b68-53e73e679e2e service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:29:54.745808 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1276028-cc7e-497e-9b68-53e73e679e2e service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:29:54.746089 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1276028-cc7e-497e-9b68-53e73e679e2e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:29:54.747691 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-c1276028-cc7e-497e-9b68-53e73e679e2e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:29:54.867558 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1276028-cc7e-497e-9b68-53e73e679e2e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:29:54.869011 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1276028-cc7e-497e-9b68-53e73e679e2e service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:29:54.888301 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 754/1509] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:29:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 348 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:29:55.269674 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a62f097-7054-484a-a714-073741f987f1 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:29:55.271006 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-2a62f097-7054-484a-a714-073741f987f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:29:55.282915 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a62f097-7054-484a-a714-073741f987f1 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:29:55.393754 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7023a12d-696a-42ee-b68d-1533a0bea3dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:29:55.394344 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7023a12d-696a-42ee-b68d-1533a0bea3dd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:29:55.394624 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7023a12d-696a-42ee-b68d-1533a0bea3dd admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:29:55.394896 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7023a12d-696a-42ee-b68d-1533a0bea3dd admin admin] PATH_INFO: `/v3/projects/5523be9f650a4e10bf46c772a891b892` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:29:55.405174 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7023a12d-696a-42ee-b68d-1533a0bea3dd admin admin] RBAC: Authorizing `identity:delete_project(project_id=5523be9f650a4e10bf46c772a891b892)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:29:55.406243 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: WARNING py.warnings [None req-7023a12d-696a-42ee-b68d-1533a0bea3dd admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 29 21:29:55.406243 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: warnings.warn(msg) Aug 29 21:29:55.406243 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: Aug 29 21:29:55.407991 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7023a12d-696a-42ee-b68d-1533a0bea3dd admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:29:55.471072 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-7023a12d-696a-42ee-b68d-1533a0bea3dd admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5523be9f650a4e10bf46c772a891b892', 'request_id': 'req-7023a12d-696a-42ee-b68d-1533a0bea3dd'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:29:55.477014 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-7023a12d-696a-42ee-b68d-1533a0bea3dd admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5523be9f650a4e10bf46c772a891b892', 'request_id': 'req-7023a12d-696a-42ee-b68d-1533a0bea3dd'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:29:55.491642 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-7023a12d-696a-42ee-b68d-1533a0bea3dd admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5523be9f650a4e10bf46c772a891b892', 'request_id': 'req-7023a12d-696a-42ee-b68d-1533a0bea3dd'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:29:55.518182 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-7023a12d-696a-42ee-b68d-1533a0bea3dd admin admin] The token cache is being invalidate because project 5523be9f650a4e10bf46c772a891b892 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81434) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:29:55.518479 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-7023a12d-696a-42ee-b68d-1533a0bea3dd admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:29:55.519008 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-7023a12d-696a-42ee-b68d-1533a0bea3dd admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:29:55.520339 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 756/1510] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:29:55 2022] DELETE /identity/v3/projects/5523be9f650a4e10bf46c772a891b892 => generated 0 bytes in 251 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:30:04.885950 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=62 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:30:05.517983 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=84,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:31:28.414764 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d968f4f0-fea6-4d8a-a2e3-b6b0406b03ab None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:31:28.415818 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d968f4f0-fea6-4d8a-a2e3-b6b0406b03ab None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:31:28.415818 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d968f4f0-fea6-4d8a-a2e3-b6b0406b03ab None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:31:28.417135 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 755/1511] 127.0.0.1 () {54 vars in 901 bytes} [Mon Aug 29 21:31:28 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:31:28.424673 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a90f3cce-63f8-466f-99f5-151563d7293a None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:31:28.425271 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a90f3cce-63f8-466f-99f5-151563d7293a None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:31:28.425602 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a90f3cce-63f8-466f-99f5-151563d7293a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:31:28.543544 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-a90f3cce-63f8-466f-99f5-151563d7293a None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:31:28.545968 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-a90f3cce-63f8-466f-99f5-151563d7293a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:31:28.694348 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 757/1512] 127.0.0.1 () {58 vars in 998 bytes} [Mon Aug 29 21:31:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 271 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:31:28.860196 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d968f4f0-fea6-4d8a-a2e3-b6b0406b03ab None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:31:28.862506 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-d968f4f0-fea6-4d8a-a2e3-b6b0406b03ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:31:29.027047 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d968f4f0-fea6-4d8a-a2e3-b6b0406b03ab None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:31:29.042226 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d785f0e-c09c-46e6-bf2e-8a60e0bc3df5 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a35b5b7dea7b421da056f063d2436ffb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:31:29.042866 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d785f0e-c09c-46e6-bf2e-8a60e0bc3df5 service nova] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:31:29.043146 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d785f0e-c09c-46e6-bf2e-8a60e0bc3df5 service nova] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:31:29.043424 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d785f0e-c09c-46e6-bf2e-8a60e0bc3df5 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:31:29.053075 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d785f0e-c09c-46e6-bf2e-8a60e0bc3df5 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:31:29.054710 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d785f0e-c09c-46e6-bf2e-8a60e0bc3df5 service nova] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:31:29.064806 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 756/1513] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:31:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 205 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:31:29.977437 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09b4d960-5a31-43c3-8d8e-65bed1d0784d None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:31:29.978396 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09b4d960-5a31-43c3-8d8e-65bed1d0784d None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:31:29.978396 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09b4d960-5a31-43c3-8d8e-65bed1d0784d None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:31:29.979367 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 758/1514] 127.0.0.1 () {54 vars in 901 bytes} [Mon Aug 29 21:31:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:31:29.986654 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c748f1e-3892-47ad-bd4d-91c61a3a8a04 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:31:29.987377 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c748f1e-3892-47ad-bd4d-91c61a3a8a04 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:31:29.987584 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c748f1e-3892-47ad-bd4d-91c61a3a8a04 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:31:30.026911 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-8c748f1e-3892-47ad-bd4d-91c61a3a8a04 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:31:30.028136 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-8c748f1e-3892-47ad-bd4d-91c61a3a8a04 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:31:30.037929 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 757/1515] 127.0.0.1 () {58 vars in 998 bytes} [Mon Aug 29 21:31:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 53 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:31:30.203158 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09b4d960-5a31-43c3-8d8e-65bed1d0784d None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:31:30.225471 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09b4d960-5a31-43c3-8d8e-65bed1d0784d None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:31:30.232522 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e965355-6c0f-4b9d-867c-2ddecab08871 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a35b5b7dea7b421da056f063d2436ffb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:31:30.233141 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e965355-6c0f-4b9d-867c-2ddecab08871 service nova] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:31:30.233418 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e965355-6c0f-4b9d-867c-2ddecab08871 service nova] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:31:30.233734 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e965355-6c0f-4b9d-867c-2ddecab08871 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:31:30.242726 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e965355-6c0f-4b9d-867c-2ddecab08871 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:31:30.244331 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e965355-6c0f-4b9d-867c-2ddecab08871 service nova] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:31:30.258175 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 759/1516] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:31:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 56 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:31:30.536072 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f907c4d-cb7d-4b0c-a560-6709dd25a65f None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:31:30.536553 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f907c4d-cb7d-4b0c-a560-6709dd25a65f None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:31:30.536988 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f907c4d-cb7d-4b0c-a560-6709dd25a65f None None] PATH_INFO: `/v3` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:31:30.539198 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 758/1517] 10.222.0.13 () {54 vars in 925 bytes} [Mon Aug 29 21:31:30 2022] GET /identity/v3 => generated 255 bytes in 5 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:31:30.547126 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c4645f7-3de1-4d2f-ade4-37b37da6dd1c None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:31:30.547591 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c4645f7-3de1-4d2f-ade4-37b37da6dd1c None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:31:30.547879 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c4645f7-3de1-4d2f-ade4-37b37da6dd1c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:31:30.633322 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-0c4645f7-3de1-4d2f-ade4-37b37da6dd1c None None] MFA Rules not processed for user `b95fa07a7c0549dbaac49406e2639e21`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:31:30.635425 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-0c4645f7-3de1-4d2f-ade4-37b37da6dd1c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:31:30.656737 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:31:30.721806 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 760/1518] 10.222.0.13 () {58 vars in 1015 bytes} [Mon Aug 29 21:31:30 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 21:31:34.651980 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cda4163e-d690-4a7f-b4c3-2fd7363212bd None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:31:34.652560 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cda4163e-d690-4a7f-b4c3-2fd7363212bd None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:31:34.652560 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cda4163e-d690-4a7f-b4c3-2fd7363212bd None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:31:34.654003 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 759/1519] 10.222.0.90 () {54 vars in 961 bytes} [Mon Aug 29 21:31:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:31:34.659317 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9040a5d-d235-4a7b-8b85-e36835148e1d None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:31:34.660003 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9040a5d-d235-4a7b-8b85-e36835148e1d None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:31:34.660003 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9040a5d-d235-4a7b-8b85-e36835148e1d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:31:34.689646 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-a9040a5d-d235-4a7b-8b85-e36835148e1d None None] MFA Rules not processed for user `b95fa07a7c0549dbaac49406e2639e21`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:31:34.690938 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-a9040a5d-d235-4a7b-8b85-e36835148e1d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:31:34.700009 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 761/1520] 10.222.0.90 () {58 vars in 1058 bytes} [Mon Aug 29 21:31:34 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:31:34.708739 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cda4163e-d690-4a7f-b4c3-2fd7363212bd None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:31:34.725606 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cda4163e-d690-4a7f-b4c3-2fd7363212bd None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:31:34.730757 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8812e543-0560-43c7-aa19-524dfd7cbf38 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b95fa07a7c0549dbaac49406e2639e21', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:31:34.731362 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8812e543-0560-43c7-aa19-524dfd7cbf38 service placement] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:31:34.731654 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8812e543-0560-43c7-aa19-524dfd7cbf38 service placement] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:31:34.732014 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8812e543-0560-43c7-aa19-524dfd7cbf38 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:31:34.741646 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8812e543-0560-43c7-aa19-524dfd7cbf38 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:31:34.743274 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8812e543-0560-43c7-aa19-524dfd7cbf38 service placement] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:31:34.755579 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 760/1521] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:31:34 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 48 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:31:41.585430 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af6d1809-c896-456d-9d1f-0ce66b080c20 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:31:41.586185 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af6d1809-c896-456d-9d1f-0ce66b080c20 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:31:41.586185 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af6d1809-c896-456d-9d1f-0ce66b080c20 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:31:41.587395 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 762/1522] 127.0.0.1 () {54 vars in 901 bytes} [Mon Aug 29 21:31:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:31:41.592906 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74ab5d84-5dca-44d5-ad68-0ff423c384e4 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:31:41.593412 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74ab5d84-5dca-44d5-ad68-0ff423c384e4 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:31:41.593509 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74ab5d84-5dca-44d5-ad68-0ff423c384e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:31:41.616272 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-74ab5d84-5dca-44d5-ad68-0ff423c384e4 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:31:41.617522 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-74ab5d84-5dca-44d5-ad68-0ff423c384e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:31:41.626816 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 761/1523] 127.0.0.1 () {58 vars in 998 bytes} [Mon Aug 29 21:31:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:31:41.795182 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af6d1809-c896-456d-9d1f-0ce66b080c20 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:31:41.811399 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af6d1809-c896-456d-9d1f-0ce66b080c20 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:31:41.818173 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23ed4e72-2af0-4057-9d12-cba34a0f4e2c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a35b5b7dea7b421da056f063d2436ffb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:31:41.818946 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23ed4e72-2af0-4057-9d12-cba34a0f4e2c service nova] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:31:41.819461 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23ed4e72-2af0-4057-9d12-cba34a0f4e2c service nova] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:31:41.819971 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23ed4e72-2af0-4057-9d12-cba34a0f4e2c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:31:41.831829 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23ed4e72-2af0-4057-9d12-cba34a0f4e2c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:31:41.836508 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23ed4e72-2af0-4057-9d12-cba34a0f4e2c service nova] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:31:41.849522 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 763/1524] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:31:41 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 55 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:31:42.792559 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fac90ef-2e0b-4e80-9345-3a70355b85f4 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:31:42.793315 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fac90ef-2e0b-4e80-9345-3a70355b85f4 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:31:42.793315 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fac90ef-2e0b-4e80-9345-3a70355b85f4 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:31:42.794989 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 762/1525] 127.0.0.1 () {54 vars in 901 bytes} [Mon Aug 29 21:31:42 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 29 21:31:42.802171 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75af476f-f5f8-44a5-8de4-cbffa2d797f7 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:31:42.802987 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75af476f-f5f8-44a5-8de4-cbffa2d797f7 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:31:42.802987 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75af476f-f5f8-44a5-8de4-cbffa2d797f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:31:42.829860 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-75af476f-f5f8-44a5-8de4-cbffa2d797f7 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:31:42.831120 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-75af476f-f5f8-44a5-8de4-cbffa2d797f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:31:42.839991 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 764/1526] 127.0.0.1 () {58 vars in 998 bytes} [Mon Aug 29 21:31:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:31:43.023045 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fac90ef-2e0b-4e80-9345-3a70355b85f4 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:31:43.042487 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fac90ef-2e0b-4e80-9345-3a70355b85f4 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:31:43.051342 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2e84b59-1c52-49c6-978c-80785012996d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a35b5b7dea7b421da056f063d2436ffb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:31:43.052058 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2e84b59-1c52-49c6-978c-80785012996d service nova] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:31:43.052419 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2e84b59-1c52-49c6-978c-80785012996d service nova] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:31:43.052739 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2e84b59-1c52-49c6-978c-80785012996d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:31:43.062656 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2e84b59-1c52-49c6-978c-80785012996d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:31:43.064273 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2e84b59-1c52-49c6-978c-80785012996d service nova] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:31:43.075119 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 763/1527] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:31:43 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 53 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:31:47.627326 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5afe68da-21a0-43c8-b5f9-add5ce932a2d None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:31:47.627909 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5afe68da-21a0-43c8-b5f9-add5ce932a2d None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:31:47.627909 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5afe68da-21a0-43c8-b5f9-add5ce932a2d None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:31:47.629268 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 765/1528] 127.0.0.1 () {54 vars in 917 bytes} [Mon Aug 29 21:31:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:31:47.635336 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56395c5a-c9a0-4fb0-84bc-d1585e18cf5e None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:31:47.636059 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56395c5a-c9a0-4fb0-84bc-d1585e18cf5e None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:31:47.636059 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56395c5a-c9a0-4fb0-84bc-d1585e18cf5e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:31:47.657708 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-56395c5a-c9a0-4fb0-84bc-d1585e18cf5e None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:31:47.659157 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-56395c5a-c9a0-4fb0-84bc-d1585e18cf5e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:31:47.669914 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 764/1529] 127.0.0.1 () {58 vars in 1014 bytes} [Mon Aug 29 21:31:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 21:31:47.993464 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5afe68da-21a0-43c8-b5f9-add5ce932a2d None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:31:47.995681 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-5afe68da-21a0-43c8-b5f9-add5ce932a2d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:31:48.100128 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5afe68da-21a0-43c8-b5f9-add5ce932a2d None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:31:48.104729 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbad7b73-83d5-47f7-abb0-823cc393c60a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:31:48.105349 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbad7b73-83d5-47f7-abb0-823cc393c60a service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:31:48.105634 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbad7b73-83d5-47f7-abb0-823cc393c60a service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:31:48.106001 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbad7b73-83d5-47f7-abb0-823cc393c60a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:31:48.116691 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbad7b73-83d5-47f7-abb0-823cc393c60a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:31:48.118286 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbad7b73-83d5-47f7-abb0-823cc393c60a service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:31:48.129687 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 766/1530] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:31:47 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 141 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:31:50.177583 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc53d8d4-07a0-4a0b-af17-9536520505d7 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:31:50.178176 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc53d8d4-07a0-4a0b-af17-9536520505d7 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:31:50.178176 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc53d8d4-07a0-4a0b-af17-9536520505d7 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:31:50.179570 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 765/1531] 127.0.0.1 () {54 vars in 917 bytes} [Mon Aug 29 21:31:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:31:50.189193 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3756bda5-14c1-431d-af7d-604a0b738567 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:31:50.189772 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3756bda5-14c1-431d-af7d-604a0b738567 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:31:50.189950 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3756bda5-14c1-431d-af7d-604a0b738567 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:31:50.215468 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-3756bda5-14c1-431d-af7d-604a0b738567 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:31:50.216702 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-3756bda5-14c1-431d-af7d-604a0b738567 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:31:50.224839 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 767/1532] 127.0.0.1 () {58 vars in 1014 bytes} [Mon Aug 29 21:31:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:31:50.530856 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc53d8d4-07a0-4a0b-af17-9536520505d7 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:31:50.543053 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc53d8d4-07a0-4a0b-af17-9536520505d7 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:31:50.547304 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5c24fac-3526-4032-b657-c46dbc7429aa service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:31:50.547918 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5c24fac-3526-4032-b657-c46dbc7429aa service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:31:50.548216 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5c24fac-3526-4032-b657-c46dbc7429aa service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:31:50.548486 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5c24fac-3526-4032-b657-c46dbc7429aa service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:31:50.560625 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5c24fac-3526-4032-b657-c46dbc7429aa service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:31:50.562946 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5c24fac-3526-4032-b657-c46dbc7429aa service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:31:50.575485 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 766/1533] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:31:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:31:54.178536 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffeda5a3-c16f-48d4-95a5-c01f9ff50cbf None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:31:54.179112 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffeda5a3-c16f-48d4-95a5-c01f9ff50cbf None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:31:54.179112 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffeda5a3-c16f-48d4-95a5-c01f9ff50cbf None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:31:54.180451 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 768/1534] 127.0.0.1 () {54 vars in 917 bytes} [Mon Aug 29 21:31:54 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:31:54.186554 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3b2f508-ebdb-4353-a11b-3d6db5a52f4a None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:31:54.187085 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3b2f508-ebdb-4353-a11b-3d6db5a52f4a None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:31:54.187085 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3b2f508-ebdb-4353-a11b-3d6db5a52f4a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:31:54.207952 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-f3b2f508-ebdb-4353-a11b-3d6db5a52f4a None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:31:54.209172 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-f3b2f508-ebdb-4353-a11b-3d6db5a52f4a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:31:54.218959 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 767/1535] 127.0.0.1 () {58 vars in 1014 bytes} [Mon Aug 29 21:31:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:31:54.796052 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffeda5a3-c16f-48d4-95a5-c01f9ff50cbf None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:31:54.797885 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-ffeda5a3-c16f-48d4-95a5-c01f9ff50cbf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:31:54.884038 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:31:54.899388 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffeda5a3-c16f-48d4-95a5-c01f9ff50cbf None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:31:54.904060 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3382614f-33f8-449e-ad61-d6477c93fc05 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dc226f7ddcb49ecacb51ce57fdfc11b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:31:54.904894 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3382614f-33f8-449e-ad61-d6477c93fc05 service cinder] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:31:54.905175 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3382614f-33f8-449e-ad61-d6477c93fc05 service cinder] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:31:54.905451 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3382614f-33f8-449e-ad61-d6477c93fc05 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:31:54.915724 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3382614f-33f8-449e-ad61-d6477c93fc05 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:31:54.917291 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3382614f-33f8-449e-ad61-d6477c93fc05 service cinder] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:31:54.943679 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 769/1536] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:31:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 148 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:31:57.015542 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b31d1614-ff23-47b2-b571-c4137a26e6cb None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:31:57.016290 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b31d1614-ff23-47b2-b571-c4137a26e6cb None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:31:57.016290 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b31d1614-ff23-47b2-b571-c4137a26e6cb None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:31:57.017603 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 768/1537] 127.0.0.1 () {54 vars in 917 bytes} [Mon Aug 29 21:31:57 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:31:57.036568 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-116ff426-2121-4c50-aa8b-6daa4d5be7b1 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:31:57.037390 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-116ff426-2121-4c50-aa8b-6daa4d5be7b1 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:31:57.037557 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-116ff426-2121-4c50-aa8b-6daa4d5be7b1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:31:57.064071 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-116ff426-2121-4c50-aa8b-6daa4d5be7b1 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:31:57.065282 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-116ff426-2121-4c50-aa8b-6daa4d5be7b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:31:57.073942 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 770/1538] 127.0.0.1 () {58 vars in 1014 bytes} [Mon Aug 29 21:31:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:31:57.667595 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b31d1614-ff23-47b2-b571-c4137a26e6cb None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:31:57.669580 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-b31d1614-ff23-47b2-b571-c4137a26e6cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:31:57.679482 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=84 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:31:57.688838 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b31d1614-ff23-47b2-b571-c4137a26e6cb None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:31:57.692756 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81c4e38f-4943-48f9-91a2-78fc58953a70 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dc226f7ddcb49ecacb51ce57fdfc11b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:31:57.693377 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c4e38f-4943-48f9-91a2-78fc58953a70 service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:31:57.693716 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c4e38f-4943-48f9-91a2-78fc58953a70 service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:31:57.694024 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c4e38f-4943-48f9-91a2-78fc58953a70 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:31:57.705620 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81c4e38f-4943-48f9-91a2-78fc58953a70 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:31:57.707101 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81c4e38f-4943-48f9-91a2-78fc58953a70 service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:31:57.720295 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 769/1539] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:31:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 53 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:32:07.058363 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=11 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:32:07.713171 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=5 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:33:27.489135 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-125d16a4-5cf5-4316-bbe6-ea43f71dfe2c None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:27.489976 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-125d16a4-5cf5-4316-bbe6-ea43f71dfe2c None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:27.489976 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-125d16a4-5cf5-4316-bbe6-ea43f71dfe2c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:27.539347 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-125d16a4-5cf5-4316-bbe6-ea43f71dfe2c None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:33:27.541393 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-125d16a4-5cf5-4316-bbe6-ea43f71dfe2c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:33:27.561966 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 771/1540] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:33:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 74 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:33:27.572790 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-502685e6-452a-4d49-ae2a-d986c0177b8b None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:27.573941 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-502685e6-452a-4d49-ae2a-d986c0177b8b None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:27.574459 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-502685e6-452a-4d49-ae2a-d986c0177b8b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:27.605414 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-502685e6-452a-4d49-ae2a-d986c0177b8b None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:33:27.607191 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-502685e6-452a-4d49-ae2a-d986c0177b8b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:33:27.623640 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 770/1541] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:33:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 53 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 21:33:27.630264 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-125d16a4-5cf5-4316-bbe6-ea43f71dfe2c None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:27.655453 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-125d16a4-5cf5-4316-bbe6-ea43f71dfe2c None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:27.667279 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8074d57e-5dda-47e7-8989-0dc3a415514a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:27.668154 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8074d57e-5dda-47e7-8989-0dc3a415514a admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:27.668439 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8074d57e-5dda-47e7-8989-0dc3a415514a admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:27.668714 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8074d57e-5dda-47e7-8989-0dc3a415514a admin admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:27.669468 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8074d57e-5dda-47e7-8989-0dc3a415514a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:33:27.669987 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8074d57e-5dda-47e7-8989-0dc3a415514a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:27.672721 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8074d57e-5dda-47e7-8989-0dc3a415514a admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:27.693887 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 772/1542] 10.222.0.90 () {58 vars in 1153 bytes} [Mon Aug 29 21:33:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 65 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:33:27.705621 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-502685e6-452a-4d49-ae2a-d986c0177b8b None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:27.722383 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-502685e6-452a-4d49-ae2a-d986c0177b8b None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:27.731092 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a3b1cc7-82a0-4730-a709-51d033f92842 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:27.732186 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a3b1cc7-82a0-4730-a709-51d033f92842 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:27.732692 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a3b1cc7-82a0-4730-a709-51d033f92842 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:27.733285 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a3b1cc7-82a0-4730-a709-51d033f92842 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:27.734464 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a3b1cc7-82a0-4730-a709-51d033f92842 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:27.736647 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a3b1cc7-82a0-4730-a709-51d033f92842 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:27.807855 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 771/1543] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:33:27 2022] POST /identity/v3/projects => generated 393 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:33:27.813610 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8074d57e-5dda-47e7-8989-0dc3a415514a admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:27.842331 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8074d57e-5dda-47e7-8989-0dc3a415514a admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:27.986512 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7938af44-1e01-49a4-8302-f17836d20f0e None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:27.986858 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7938af44-1e01-49a4-8302-f17836d20f0e None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:27.987846 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7938af44-1e01-49a4-8302-f17836d20f0e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:28.041815 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-7938af44-1e01-49a4-8302-f17836d20f0e None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:33:28.043544 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4fa0e7a-324a-44d3-85a1-282f7fb9efc9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:28.044113 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-7938af44-1e01-49a4-8302-f17836d20f0e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:33:28.044747 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4fa0e7a-324a-44d3-85a1-282f7fb9efc9 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:28.045301 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4fa0e7a-324a-44d3-85a1-282f7fb9efc9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:28.045818 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4fa0e7a-324a-44d3-85a1-282f7fb9efc9 admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:28.046701 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4fa0e7a-324a-44d3-85a1-282f7fb9efc9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:28.048989 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4fa0e7a-324a-44d3-85a1-282f7fb9efc9 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:28.065261 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 772/1544] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:33:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 21:33:28.091727 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f781f0bb-a7aa-4b27-a7d9-d6c908f3bba1 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:28.092071 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f781f0bb-a7aa-4b27-a7d9-d6c908f3bba1 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:28.092367 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f781f0bb-a7aa-4b27-a7d9-d6c908f3bba1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:28.095388 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 773/1545] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:33:27 2022] POST /identity/v3/users => generated 297 bytes in 283 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:33:28.103768 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4fa0e7a-324a-44d3-85a1-282f7fb9efc9 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:28.130648 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4fa0e7a-324a-44d3-85a1-282f7fb9efc9 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:28.143469 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fea7a1f-7e5c-48f1-8002-9fac1650c3df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:28.150385 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-f781f0bb-a7aa-4b27-a7d9-d6c908f3bba1 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:33:28.150385 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-f781f0bb-a7aa-4b27-a7d9-d6c908f3bba1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:33:28.152462 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fea7a1f-7e5c-48f1-8002-9fac1650c3df admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:28.153122 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fea7a1f-7e5c-48f1-8002-9fac1650c3df admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:28.153811 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fea7a1f-7e5c-48f1-8002-9fac1650c3df admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:28.155317 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fea7a1f-7e5c-48f1-8002-9fac1650c3df admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:28.158076 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fea7a1f-7e5c-48f1-8002-9fac1650c3df admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:28.163563 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 773/1546] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:33:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 75 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 21:33:28.171252 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f781f0bb-a7aa-4b27-a7d9-d6c908f3bba1 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:28.180524 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 774/1547] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:33:28 2022] GET /identity/v3/roles => generated 1420 bytes in 78 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 29 21:33:28.185764 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f781f0bb-a7aa-4b27-a7d9-d6c908f3bba1 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:28.194000 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fea7a1f-7e5c-48f1-8002-9fac1650c3df admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:28.201916 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d71f3975-68b1-4217-83ce-ceeae9d5becc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:28.202561 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d71f3975-68b1-4217-83ce-ceeae9d5becc admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:28.202839 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d71f3975-68b1-4217-83ce-ceeae9d5becc admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:28.203125 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d71f3975-68b1-4217-83ce-ceeae9d5becc admin admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:28.203815 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d71f3975-68b1-4217-83ce-ceeae9d5becc admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:33:28.204116 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d71f3975-68b1-4217-83ce-ceeae9d5becc admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:28.205213 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: WARNING py.warnings [None req-d71f3975-68b1-4217-83ce-ceeae9d5becc admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Aug 29 21:33:28.205213 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: warnings.warn(msg) Aug 29 21:33:28.205213 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: Aug 29 21:33:28.209179 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d71f3975-68b1-4217-83ce-ceeae9d5becc admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:28.223538 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 774/1548] 10.222.0.90 () {58 vars in 1153 bytes} [Mon Aug 29 21:33:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:33:28.224665 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fea7a1f-7e5c-48f1-8002-9fac1650c3df admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:28.225299 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08ce51fa-213a-4913-9d86-4467216cdacb None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:28.225586 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08ce51fa-213a-4913-9d86-4467216cdacb None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:28.225974 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08ce51fa-213a-4913-9d86-4467216cdacb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:28.237612 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-244e9a57-686f-471c-81b8-fbb11b549288 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:28.238584 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-244e9a57-686f-471c-81b8-fbb11b549288 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:28.239102 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-244e9a57-686f-471c-81b8-fbb11b549288 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:28.239585 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-244e9a57-686f-471c-81b8-fbb11b549288 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:28.240915 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-244e9a57-686f-471c-81b8-fbb11b549288 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:28.244558 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-244e9a57-686f-471c-81b8-fbb11b549288 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:28.278005 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 775/1549] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:33:28 2022] GET /identity/v3/roles => generated 1420 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:33:28.286538 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-244e9a57-686f-471c-81b8-fbb11b549288 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:28.287153 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-08ce51fa-213a-4913-9d86-4467216cdacb None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:33:28.288595 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-08ce51fa-213a-4913-9d86-4467216cdacb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:33:28.307675 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 775/1550] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:33:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:33:28.310472 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a923eda-be4f-4284-b326-6c9ae7495735 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:28.310750 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a923eda-be4f-4284-b326-6c9ae7495735 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:28.311019 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a923eda-be4f-4284-b326-6c9ae7495735 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:28.316201 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-244e9a57-686f-471c-81b8-fbb11b549288 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:28.325835 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcd232d5-6fe0-4b36-b158-f0cd434fe24e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:28.326466 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcd232d5-6fe0-4b36-b158-f0cd434fe24e admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:28.326742 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcd232d5-6fe0-4b36-b158-f0cd434fe24e admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:28.327032 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcd232d5-6fe0-4b36-b158-f0cd434fe24e admin admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:28.328050 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcd232d5-6fe0-4b36-b158-f0cd434fe24e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:28.329161 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: WARNING py.warnings [None req-fcd232d5-6fe0-4b36-b158-f0cd434fe24e admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 29 21:33:28.329161 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: warnings.warn(msg) Aug 29 21:33:28.329161 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: Aug 29 21:33:28.332524 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcd232d5-6fe0-4b36-b158-f0cd434fe24e admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:28.357202 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-5a923eda-be4f-4284-b326-6c9ae7495735 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:33:28.359625 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-5a923eda-be4f-4284-b326-6c9ae7495735 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:33:28.378022 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 776/1551] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:33:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 70 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:33:28.379721 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a923eda-be4f-4284-b326-6c9ae7495735 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:28.395004 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 776/1552] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:33:28 2022] POST /identity/v3/projects => generated 377 bytes in 116 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:33:28.396503 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99b69f32-d42b-4aca-bee7-de6cd4aedbba None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:28.396801 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99b69f32-d42b-4aca-bee7-de6cd4aedbba None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:28.397092 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99b69f32-d42b-4aca-bee7-de6cd4aedbba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:28.401905 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a923eda-be4f-4284-b326-6c9ae7495735 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:28.431697 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-99b69f32-d42b-4aca-bee7-de6cd4aedbba None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:33:28.433423 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-99b69f32-d42b-4aca-bee7-de6cd4aedbba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:33:28.522800 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d511ec51-9c4a-455c-b269-43854cc8230b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:28.523652 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d511ec51-9c4a-455c-b269-43854cc8230b admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:28.524132 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d511ec51-9c4a-455c-b269-43854cc8230b admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:28.524512 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d511ec51-9c4a-455c-b269-43854cc8230b admin admin] PATH_INFO: `/v3/projects/fb0643eb366140d59f96d86259557915/users/39f161e61a514d94b6008bd006cda5ce/roles/02e242d5520740d4b77a5f4008c7eb57` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:28.534072 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 777/1553] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:33:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:33:28.535597 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad6e68de-c037-4cc3-9587-a6e40a97cb0f None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:28.535886 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad6e68de-c037-4cc3-9587-a6e40a97cb0f None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:28.536164 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad6e68de-c037-4cc3-9587-a6e40a97cb0f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:28.549600 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d511ec51-9c4a-455c-b269-43854cc8230b admin admin] RBAC: Authorizing `identity:create_grant(project_id=fb0643eb366140d59f96d86259557915, user_id=39f161e61a514d94b6008bd006cda5ce, role_id=02e242d5520740d4b77a5f4008c7eb57)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:28.555005 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d511ec51-9c4a-455c-b269-43854cc8230b admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:28.558745 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-ad6e68de-c037-4cc3-9587-a6e40a97cb0f None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:33:28.559983 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-ad6e68de-c037-4cc3-9587-a6e40a97cb0f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:33:28.569271 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 778/1554] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:33:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:33:28.570580 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d36ce476-dcfb-4714-a67e-9c8c8ecfef43 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:28.570923 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d36ce476-dcfb-4714-a67e-9c8c8ecfef43 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:28.574238 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d36ce476-dcfb-4714-a67e-9c8c8ecfef43 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:28.581282 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 777/1555] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:33:28 2022] PUT /identity/v3/projects/fb0643eb366140d59f96d86259557915/users/39f161e61a514d94b6008bd006cda5ce/roles/02e242d5520740d4b77a5f4008c7eb57 => generated 0 bytes in 198 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:33:28.581282 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efce6792-29b6-498c-ac12-0439e25c3049 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:28.581282 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efce6792-29b6-498c-ac12-0439e25c3049 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:28.581282 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efce6792-29b6-498c-ac12-0439e25c3049 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:28.604064 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-d36ce476-dcfb-4714-a67e-9c8c8ecfef43 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:33:28.605394 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-d36ce476-dcfb-4714-a67e-9c8c8ecfef43 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:33:28.610212 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-efce6792-29b6-498c-ac12-0439e25c3049 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:33:28.611448 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-efce6792-29b6-498c-ac12-0439e25c3049 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:33:28.620001 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:33:28.683048 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 779/1556] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:33:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:33:28.684213 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d36ce476-dcfb-4714-a67e-9c8c8ecfef43 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:28.684694 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 778/1557] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:33:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:33:28.686083 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-036c5bfa-2b92-412e-8bd3-31deea8cbf73 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:28.686605 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-036c5bfa-2b92-412e-8bd3-31deea8cbf73 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:28.686932 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-036c5bfa-2b92-412e-8bd3-31deea8cbf73 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:28.703000 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:33:28.710184 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d36ce476-dcfb-4714-a67e-9c8c8ecfef43 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:28.713953 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-036c5bfa-2b92-412e-8bd3-31deea8cbf73 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:33:28.715202 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-036c5bfa-2b92-412e-8bd3-31deea8cbf73 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:33:28.716275 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d95f9d0-d223-48bd-bc28-ebf2b7e90230 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:28.716896 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d95f9d0-d223-48bd-bc28-ebf2b7e90230 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:28.717177 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d95f9d0-d223-48bd-bc28-ebf2b7e90230 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:28.717447 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d95f9d0-d223-48bd-bc28-ebf2b7e90230 admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:28.718325 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d95f9d0-d223-48bd-bc28-ebf2b7e90230 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:28.720891 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d95f9d0-d223-48bd-bc28-ebf2b7e90230 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:28.723499 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 779/1558] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:33:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:33:28.724805 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca98cb8-ad92-495b-8f72-f839e7d1a636 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:28.725115 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca98cb8-ad92-495b-8f72-f839e7d1a636 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:28.725397 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca98cb8-ad92-495b-8f72-f839e7d1a636 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:28.756624 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 780/1559] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:33:28 2022] POST /identity/v3/users => generated 289 bytes in 67 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:33:28.756624 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e842eba-d9a5-483e-ab11-cd5d11153bf1 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:28.756624 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e842eba-d9a5-483e-ab11-cd5d11153bf1 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:28.756624 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e842eba-d9a5-483e-ab11-cd5d11153bf1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:28.757444 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-0ca98cb8-ad92-495b-8f72-f839e7d1a636 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:33:28.759147 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-0ca98cb8-ad92-495b-8f72-f839e7d1a636 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:33:28.780893 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 780/1560] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:33:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 57 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:33:28.782253 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22dbe334-80cb-4ccf-8042-9a76ff674d4c None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:28.782540 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22dbe334-80cb-4ccf-8042-9a76ff674d4c None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:28.782812 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22dbe334-80cb-4ccf-8042-9a76ff674d4c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:28.786433 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-5e842eba-d9a5-483e-ab11-cd5d11153bf1 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:33:28.787798 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-5e842eba-d9a5-483e-ab11-cd5d11153bf1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:33:28.797324 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 781/1561] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:33:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 45 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:33:28.797820 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-613ad839-a583-4912-8709-35812a7366be None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:28.798136 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-613ad839-a583-4912-8709-35812a7366be None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:28.798411 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-613ad839-a583-4912-8709-35812a7366be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:28.813169 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-22dbe334-80cb-4ccf-8042-9a76ff674d4c None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:33:28.814993 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-22dbe334-80cb-4ccf-8042-9a76ff674d4c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:33:28.821419 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-613ad839-a583-4912-8709-35812a7366be None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:33:28.822793 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-613ad839-a583-4912-8709-35812a7366be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:33:28.834614 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 781/1562] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:33:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 49 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:33:28.834614 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22dbe334-80cb-4ccf-8042-9a76ff674d4c None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:28.843738 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22dbe334-80cb-4ccf-8042-9a76ff674d4c None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:28.844326 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 782/1563] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:33:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 47 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:33:28.849475 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-613ad839-a583-4912-8709-35812a7366be None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:28.850041 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07e34968-1bd2-451c-80dc-1e9dc628c781 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:28.850896 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07e34968-1bd2-451c-80dc-1e9dc628c781 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:28.851005 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07e34968-1bd2-451c-80dc-1e9dc628c781 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:28.851250 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07e34968-1bd2-451c-80dc-1e9dc628c781 admin admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:28.851962 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07e34968-1bd2-451c-80dc-1e9dc628c781 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:33:28.853244 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07e34968-1bd2-451c-80dc-1e9dc628c781 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:28.855135 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07e34968-1bd2-451c-80dc-1e9dc628c781 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:28.861619 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-613ad839-a583-4912-8709-35812a7366be None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:28.870719 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c577646-7e11-4449-8628-0db284792f42 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:28.871593 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c577646-7e11-4449-8628-0db284792f42 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:28.871593 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c577646-7e11-4449-8628-0db284792f42 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:28.872032 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c577646-7e11-4449-8628-0db284792f42 admin admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:28.872351 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c577646-7e11-4449-8628-0db284792f42 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:33:28.872781 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 782/1564] 10.222.0.90 () {58 vars in 1153 bytes} [Mon Aug 29 21:33:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:33:28.873065 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c577646-7e11-4449-8628-0db284792f42 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:28.873993 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cb9aada-3d78-48c3-a56b-85467116c4cb None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:28.874342 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cb9aada-3d78-48c3-a56b-85467116c4cb None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:28.874503 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c577646-7e11-4449-8628-0db284792f42 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:28.883043 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cb9aada-3d78-48c3-a56b-85467116c4cb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:28.888608 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 783/1565] 10.222.0.90 () {58 vars in 1153 bytes} [Mon Aug 29 21:33:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:33:28.889512 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c577646-7e11-4449-8628-0db284792f42 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:28.902349 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c577646-7e11-4449-8628-0db284792f42 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:28.907458 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e56bc986-96a3-4205-8a7d-a63d7ec30076 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:28.908195 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e56bc986-96a3-4205-8a7d-a63d7ec30076 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:28.908487 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e56bc986-96a3-4205-8a7d-a63d7ec30076 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:28.908762 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e56bc986-96a3-4205-8a7d-a63d7ec30076 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:28.909413 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e56bc986-96a3-4205-8a7d-a63d7ec30076 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:28.911074 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e56bc986-96a3-4205-8a7d-a63d7ec30076 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:28.920575 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-9cb9aada-3d78-48c3-a56b-85467116c4cb None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:33:28.921990 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-9cb9aada-3d78-48c3-a56b-85467116c4cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:33:28.923886 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 784/1566] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:33:28 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:33:28.925273 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8494dd5-db82-499a-98a5-5f573f1ac6b4 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:28.925561 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8494dd5-db82-499a-98a5-5f573f1ac6b4 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:28.925885 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8494dd5-db82-499a-98a5-5f573f1ac6b4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:28.940222 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 783/1567] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:33:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 67 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:33:28.942055 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cb9aada-3d78-48c3-a56b-85467116c4cb None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:28.953649 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-b8494dd5-db82-499a-98a5-5f573f1ac6b4 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:33:28.954154 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cb9aada-3d78-48c3-a56b-85467116c4cb None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:28.954905 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-b8494dd5-db82-499a-98a5-5f573f1ac6b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:33:28.959222 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-003a811d-8669-4d25-8d3e-2330ec4e0b1d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:28.959874 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-003a811d-8669-4d25-8d3e-2330ec4e0b1d admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:28.960153 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-003a811d-8669-4d25-8d3e-2330ec4e0b1d admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:28.960426 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-003a811d-8669-4d25-8d3e-2330ec4e0b1d admin admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:28.961090 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-003a811d-8669-4d25-8d3e-2330ec4e0b1d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:33:28.962297 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-003a811d-8669-4d25-8d3e-2330ec4e0b1d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:28.964277 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-003a811d-8669-4d25-8d3e-2330ec4e0b1d admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:28.964767 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 785/1568] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:33:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:33:28.965513 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8494dd5-db82-499a-98a5-5f573f1ac6b4 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:28.975698 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8494dd5-db82-499a-98a5-5f573f1ac6b4 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:28.979619 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 784/1569] 10.222.0.90 () {58 vars in 1153 bytes} [Mon Aug 29 21:33:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:33:28.981372 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-003a811d-8669-4d25-8d3e-2330ec4e0b1d admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:28.989331 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90b8a2fa-1f8a-4b04-8109-3badbe923862 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:28.990084 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90b8a2fa-1f8a-4b04-8109-3badbe923862 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:28.990362 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90b8a2fa-1f8a-4b04-8109-3badbe923862 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:28.991655 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90b8a2fa-1f8a-4b04-8109-3badbe923862 admin admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:28.991655 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90b8a2fa-1f8a-4b04-8109-3badbe923862 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:33:28.992081 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90b8a2fa-1f8a-4b04-8109-3badbe923862 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:28.994651 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90b8a2fa-1f8a-4b04-8109-3badbe923862 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:29.013770 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 786/1570] 10.222.0.90 () {58 vars in 1153 bytes} [Mon Aug 29 21:33:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:33:29.015727 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90b8a2fa-1f8a-4b04-8109-3badbe923862 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:29.035541 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90b8a2fa-1f8a-4b04-8109-3badbe923862 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:29.043270 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae6e24e8-1d90-41f4-8b80-e70a7146e8f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:29.044320 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae6e24e8-1d90-41f4-8b80-e70a7146e8f3 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:29.044720 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae6e24e8-1d90-41f4-8b80-e70a7146e8f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:29.045225 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae6e24e8-1d90-41f4-8b80-e70a7146e8f3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:29.046291 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae6e24e8-1d90-41f4-8b80-e70a7146e8f3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:29.048336 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae6e24e8-1d90-41f4-8b80-e70a7146e8f3 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:29.058889 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-003a811d-8669-4d25-8d3e-2330ec4e0b1d admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:29.065083 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a32ec288-576d-40a2-93a3-afe965dff791 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:29.065853 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a32ec288-576d-40a2-93a3-afe965dff791 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:29.066136 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a32ec288-576d-40a2-93a3-afe965dff791 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:29.066410 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a32ec288-576d-40a2-93a3-afe965dff791 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:29.079505 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a32ec288-576d-40a2-93a3-afe965dff791 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:29.081161 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a32ec288-576d-40a2-93a3-afe965dff791 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:29.087007 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 787/1571] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:33:29 2022] POST /identity/v3/projects => generated 383 bytes in 72 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:33:29.087962 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae6e24e8-1d90-41f4-8b80-e70a7146e8f3 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:29.094500 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 785/1572] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:33:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 114 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:33:29.095405 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a32ec288-576d-40a2-93a3-afe965dff791 service neutron] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:29.110018 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a32ec288-576d-40a2-93a3-afe965dff791 service neutron] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:29.221834 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-719c045f-180b-42c9-8e60-2229b19f854f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:29.223298 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-719c045f-180b-42c9-8e60-2229b19f854f admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:29.223867 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-719c045f-180b-42c9-8e60-2229b19f854f admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:29.224082 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-719c045f-180b-42c9-8e60-2229b19f854f admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:29.226094 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-719c045f-180b-42c9-8e60-2229b19f854f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:29.228629 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-719c045f-180b-42c9-8e60-2229b19f854f admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:29.229679 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae6e24e8-1d90-41f4-8b80-e70a7146e8f3 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:29.240267 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47ced3c7-9c16-4451-b0ec-47b350146ef5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:29.240550 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 786/1573] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:33:29 2022] GET /identity/v3/roles => generated 1420 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:33:29.242060 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ced3c7-9c16-4451-b0ec-47b350146ef5 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:29.243257 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ced3c7-9c16-4451-b0ec-47b350146ef5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:29.244479 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ced3c7-9c16-4451-b0ec-47b350146ef5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:29.245409 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-719c045f-180b-42c9-8e60-2229b19f854f admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:29.245553 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47ced3c7-9c16-4451-b0ec-47b350146ef5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:29.249380 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47ced3c7-9c16-4451-b0ec-47b350146ef5 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:29.257760 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-719c045f-180b-42c9-8e60-2229b19f854f admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:29.264672 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eac3dade-2f53-479a-9a1b-3ab953a74594 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:29.265474 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac3dade-2f53-479a-9a1b-3ab953a74594 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:29.265813 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac3dade-2f53-479a-9a1b-3ab953a74594 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:29.266220 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac3dade-2f53-479a-9a1b-3ab953a74594 admin admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:29.267013 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eac3dade-2f53-479a-9a1b-3ab953a74594 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:33:29.267446 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eac3dade-2f53-479a-9a1b-3ab953a74594 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:29.269860 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eac3dade-2f53-479a-9a1b-3ab953a74594 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:29.293219 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 787/1574] 10.222.0.90 () {58 vars in 1153 bytes} [Mon Aug 29 21:33:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:33:29.295226 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eac3dade-2f53-479a-9a1b-3ab953a74594 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:29.319219 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eac3dade-2f53-479a-9a1b-3ab953a74594 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:29.328425 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98bfbf2b-4ba0-4f6a-abb8-f79eb2b76fec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:29.329082 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98bfbf2b-4ba0-4f6a-abb8-f79eb2b76fec admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:29.329562 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98bfbf2b-4ba0-4f6a-abb8-f79eb2b76fec admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:29.329938 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98bfbf2b-4ba0-4f6a-abb8-f79eb2b76fec admin admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:29.331139 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98bfbf2b-4ba0-4f6a-abb8-f79eb2b76fec admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:33:29.331651 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98bfbf2b-4ba0-4f6a-abb8-f79eb2b76fec admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:29.333607 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98bfbf2b-4ba0-4f6a-abb8-f79eb2b76fec admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:29.334996 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 788/1575] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:33:29 2022] POST /identity/v3/projects => generated 403 bytes in 247 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:33:29.340750 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47ced3c7-9c16-4451-b0ec-47b350146ef5 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:29.355074 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:33:29.361424 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 788/1576] 10.222.0.90 () {58 vars in 1153 bytes} [Mon Aug 29 21:33:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 67 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:33:29.362875 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98bfbf2b-4ba0-4f6a-abb8-f79eb2b76fec admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:29.385689 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98bfbf2b-4ba0-4f6a-abb8-f79eb2b76fec admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:29.434446 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:33:29.447911 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47ced3c7-9c16-4451-b0ec-47b350146ef5 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:29.453839 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6c9d10b-f668-47e2-9c57-365ba57adfdc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:29.454583 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c9d10b-f668-47e2-9c57-365ba57adfdc admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:29.454926 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c9d10b-f668-47e2-9c57-365ba57adfdc admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:29.455275 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c9d10b-f668-47e2-9c57-365ba57adfdc admin admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:29.456355 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6c9d10b-f668-47e2-9c57-365ba57adfdc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:29.458951 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6c9d10b-f668-47e2-9c57-365ba57adfdc admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:29.485056 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ccca0aa-670b-4f5a-b4a9-b88d39abe296 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:29.485683 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ccca0aa-670b-4f5a-b4a9-b88d39abe296 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:29.485999 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ccca0aa-670b-4f5a-b4a9-b88d39abe296 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:29.486278 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ccca0aa-670b-4f5a-b4a9-b88d39abe296 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:29.486926 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ccca0aa-670b-4f5a-b4a9-b88d39abe296 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:29.489216 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ccca0aa-670b-4f5a-b4a9-b88d39abe296 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:29.508532 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 789/1577] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:33:29 2022] POST /identity/v3/projects => generated 367 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:33:29.510963 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6c9d10b-f668-47e2-9c57-365ba57adfdc admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:29.543383 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 789/1578] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:33:29 2022] POST /identity/v3/projects => generated 367 bytes in 181 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:33:29.545202 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ccca0aa-670b-4f5a-b4a9-b88d39abe296 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:29.617542 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6c9d10b-f668-47e2-9c57-365ba57adfdc admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:29.663073 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ccca0aa-670b-4f5a-b4a9-b88d39abe296 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:29.669269 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec901577-5f6f-4957-9d91-9182a233725e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:29.670208 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec901577-5f6f-4957-9d91-9182a233725e admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:29.670758 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec901577-5f6f-4957-9d91-9182a233725e admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:29.671217 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec901577-5f6f-4957-9d91-9182a233725e admin admin] PATH_INFO: `/v3/projects/a2847581073e482d8ec3090bcafe93a7/users/b33a740f1c8a4d9db8d199eff66b76b2/roles/02e242d5520740d4b77a5f4008c7eb57` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:29.701235 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec901577-5f6f-4957-9d91-9182a233725e admin admin] RBAC: Authorizing `identity:create_grant(project_id=a2847581073e482d8ec3090bcafe93a7, user_id=b33a740f1c8a4d9db8d199eff66b76b2, role_id=02e242d5520740d4b77a5f4008c7eb57)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:29.705545 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec901577-5f6f-4957-9d91-9182a233725e admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:29.732243 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 790/1579] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:33:29 2022] PUT /identity/v3/projects/a2847581073e482d8ec3090bcafe93a7/users/b33a740f1c8a4d9db8d199eff66b76b2/roles/02e242d5520740d4b77a5f4008c7eb57 => generated 0 bytes in 188 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:33:29.733538 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec901577-5f6f-4957-9d91-9182a233725e admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:29.743546 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c78e0270-94b3-4488-8bd8-5d3def42ba66 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:29.745096 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c78e0270-94b3-4488-8bd8-5d3def42ba66 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:29.746367 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c78e0270-94b3-4488-8bd8-5d3def42ba66 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:29.746367 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c78e0270-94b3-4488-8bd8-5d3def42ba66 admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:29.752238 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c78e0270-94b3-4488-8bd8-5d3def42ba66 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:29.756467 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c78e0270-94b3-4488-8bd8-5d3def42ba66 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:29.783868 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 790/1580] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:33:29 2022] POST /identity/v3/users => generated 292 bytes in 275 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:33:29.785021 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c78e0270-94b3-4488-8bd8-5d3def42ba66 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:29.801154 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c78e0270-94b3-4488-8bd8-5d3def42ba66 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:29.861988 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:33:29.887777 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec901577-5f6f-4957-9d91-9182a233725e admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:29.894106 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d37c73a4-8f8d-4366-bf7a-32ce05301e37 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:29.895785 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d37c73a4-8f8d-4366-bf7a-32ce05301e37 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:29.895957 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d37c73a4-8f8d-4366-bf7a-32ce05301e37 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:29.896572 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d37c73a4-8f8d-4366-bf7a-32ce05301e37 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:29.897826 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d37c73a4-8f8d-4366-bf7a-32ce05301e37 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:29.901233 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d37c73a4-8f8d-4366-bf7a-32ce05301e37 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:29.912786 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea56641a-91b1-4798-992b-0eca76fb0eb7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:29.913495 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea56641a-91b1-4798-992b-0eca76fb0eb7 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:29.914007 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea56641a-91b1-4798-992b-0eca76fb0eb7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:29.914552 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea56641a-91b1-4798-992b-0eca76fb0eb7 admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:29.916202 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea56641a-91b1-4798-992b-0eca76fb0eb7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:29.918609 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea56641a-91b1-4798-992b-0eca76fb0eb7 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:29.952330 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 791/1581] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:33:29 2022] POST /identity/v3/projects => generated 377 bytes in 215 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:33:29.952330 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d37c73a4-8f8d-4366-bf7a-32ce05301e37 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:29.961878 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 791/1582] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:33:29 2022] POST /identity/v3/users => generated 302 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:33:29.961878 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea56641a-91b1-4798-992b-0eca76fb0eb7 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:29.972730 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea56641a-91b1-4798-992b-0eca76fb0eb7 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:29.998092 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:33:30.062142 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d364aa0f-1578-4766-b610-ee768710fba2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:30.063028 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d364aa0f-1578-4766-b610-ee768710fba2 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:30.063507 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d364aa0f-1578-4766-b610-ee768710fba2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:30.064103 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d364aa0f-1578-4766-b610-ee768710fba2 admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:30.064828 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d364aa0f-1578-4766-b610-ee768710fba2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:30.067274 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d364aa0f-1578-4766-b610-ee768710fba2 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:30.081345 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d37c73a4-8f8d-4366-bf7a-32ce05301e37 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:30.096007 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b4582f0-b0da-4292-8a6f-71c33896feba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:30.098071 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b4582f0-b0da-4292-8a6f-71c33896feba admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:30.098544 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b4582f0-b0da-4292-8a6f-71c33896feba admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:30.098993 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b4582f0-b0da-4292-8a6f-71c33896feba admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:30.100586 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b4582f0-b0da-4292-8a6f-71c33896feba admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:30.105216 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b4582f0-b0da-4292-8a6f-71c33896feba admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:30.121031 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 792/1583] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:33:29 2022] POST /identity/v3/users => generated 284 bytes in 160 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:33:30.121549 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d364aa0f-1578-4766-b610-ee768710fba2 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:30.140332 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d364aa0f-1578-4766-b610-ee768710fba2 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:30.149747 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c786042-5b06-410a-932d-a100f99b863e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:30.150847 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c786042-5b06-410a-932d-a100f99b863e admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:30.151337 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c786042-5b06-410a-932d-a100f99b863e admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:30.151823 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c786042-5b06-410a-932d-a100f99b863e admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:30.152928 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c786042-5b06-410a-932d-a100f99b863e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:30.156596 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c786042-5b06-410a-932d-a100f99b863e admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:30.161280 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 792/1584] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:33:29 2022] POST /identity/v3/projects => generated 385 bytes in 211 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:33:30.162933 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fcaf9c6-4af5-478b-b481-fbf1b37d185e None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:30.163088 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fcaf9c6-4af5-478b-b481-fbf1b37d185e None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:30.163320 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fcaf9c6-4af5-478b-b481-fbf1b37d185e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:30.204194 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 793/1585] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:33:30 2022] POST /identity/v3/users => generated 284 bytes in 85 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:33:30.205603 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c786042-5b06-410a-932d-a100f99b863e admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:30.210212 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-8fcaf9c6-4af5-478b-b481-fbf1b37d185e None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:33:30.214602 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-8fcaf9c6-4af5-478b-b481-fbf1b37d185e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:33:30.225120 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c786042-5b06-410a-932d-a100f99b863e admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:30.355005 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 793/1586] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:33:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 193 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:33:30.355768 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fcaf9c6-4af5-478b-b481-fbf1b37d185e None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:30.363443 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ad5cf49-a26b-40b3-a7e6-729b22b2cdd8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:30.364619 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ad5cf49-a26b-40b3-a7e6-729b22b2cdd8 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:30.365206 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ad5cf49-a26b-40b3-a7e6-729b22b2cdd8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:30.365777 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ad5cf49-a26b-40b3-a7e6-729b22b2cdd8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:30.367229 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ad5cf49-a26b-40b3-a7e6-729b22b2cdd8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:30.367528 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fcaf9c6-4af5-478b-b481-fbf1b37d185e None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:30.371333 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ad5cf49-a26b-40b3-a7e6-729b22b2cdd8 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:30.373530 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-495a3a60-8e47-4b3e-9be8-5a092632832f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:30.374302 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-495a3a60-8e47-4b3e-9be8-5a092632832f admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:30.374661 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-495a3a60-8e47-4b3e-9be8-5a092632832f admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:30.375208 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-495a3a60-8e47-4b3e-9be8-5a092632832f admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:30.376071 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-495a3a60-8e47-4b3e-9be8-5a092632832f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:30.377417 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: WARNING py.warnings [None req-495a3a60-8e47-4b3e-9be8-5a092632832f admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 29 21:33:30.377417 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: warnings.warn(msg) Aug 29 21:33:30.377417 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: Aug 29 21:33:30.379436 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-495a3a60-8e47-4b3e-9be8-5a092632832f admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:30.401337 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 794/1587] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:33:30 2022] GET /identity/v3/roles => generated 1420 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:33:30.403037 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ad5cf49-a26b-40b3-a7e6-729b22b2cdd8 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:30.418979 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 794/1588] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:33:30 2022] POST /identity/v3/users => generated 289 bytes in 63 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:33:30.420139 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-495a3a60-8e47-4b3e-9be8-5a092632832f admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:30.429755 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ad5cf49-a26b-40b3-a7e6-729b22b2cdd8 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:30.438842 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-495a3a60-8e47-4b3e-9be8-5a092632832f admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:30.439632 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17e6cff1-a1b0-4338-93ff-5ed723eefeaf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:30.440697 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17e6cff1-a1b0-4338-93ff-5ed723eefeaf admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:30.441192 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17e6cff1-a1b0-4338-93ff-5ed723eefeaf admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:30.441696 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17e6cff1-a1b0-4338-93ff-5ed723eefeaf admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:30.443175 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17e6cff1-a1b0-4338-93ff-5ed723eefeaf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:30.447832 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34f233f2-6215-45e9-b7ff-4f2137bb39f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:30.448111 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17e6cff1-a1b0-4338-93ff-5ed723eefeaf admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:30.449238 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34f233f2-6215-45e9-b7ff-4f2137bb39f3 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:30.450900 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34f233f2-6215-45e9-b7ff-4f2137bb39f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:30.451441 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34f233f2-6215-45e9-b7ff-4f2137bb39f3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:30.452809 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34f233f2-6215-45e9-b7ff-4f2137bb39f3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:30.455730 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34f233f2-6215-45e9-b7ff-4f2137bb39f3 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:30.470035 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 795/1589] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:33:30 2022] GET /identity/v3/roles => generated 1420 bytes in 68 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:33:30.474937 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17e6cff1-a1b0-4338-93ff-5ed723eefeaf admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:30.486419 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 795/1590] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:33:30 2022] GET /identity/v3/roles => generated 1420 bytes in 62 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:33:30.488223 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34f233f2-6215-45e9-b7ff-4f2137bb39f3 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:30.488411 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17e6cff1-a1b0-4338-93ff-5ed723eefeaf admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:30.498662 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40702c29-5961-4aad-8385-3cd165cf8db0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:30.499306 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40702c29-5961-4aad-8385-3cd165cf8db0 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:30.499913 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40702c29-5961-4aad-8385-3cd165cf8db0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:30.500200 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40702c29-5961-4aad-8385-3cd165cf8db0 admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:30.500862 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40702c29-5961-4aad-8385-3cd165cf8db0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:30.503151 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40702c29-5961-4aad-8385-3cd165cf8db0 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:30.503687 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34f233f2-6215-45e9-b7ff-4f2137bb39f3 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:30.512183 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb551f7a-576d-439f-be69-0a0b0a77a400 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:30.513643 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb551f7a-576d-439f-be69-0a0b0a77a400 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:30.514243 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb551f7a-576d-439f-be69-0a0b0a77a400 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:30.514864 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb551f7a-576d-439f-be69-0a0b0a77a400 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:30.516205 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb551f7a-576d-439f-be69-0a0b0a77a400 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:30.520673 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb551f7a-576d-439f-be69-0a0b0a77a400 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:30.524723 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=88,keystone:INSERT=12 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:33:30.539151 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 796/1591] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:33:30 2022] GET /identity/v3/roles => generated 1420 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:33:30.544812 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb551f7a-576d-439f-be69-0a0b0a77a400 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:30.548804 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 796/1592] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:33:30 2022] POST /identity/v3/users => generated 293 bytes in 77 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:33:30.549159 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-cb551f7a-576d-439f-be69-0a0b0a77a400 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:33:30.550619 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40702c29-5961-4aad-8385-3cd165cf8db0 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:30.578755 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40702c29-5961-4aad-8385-3cd165cf8db0 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:30.590352 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5fee240-4c73-4a57-9160-348fc4e2b00a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:30.592068 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5fee240-4c73-4a57-9160-348fc4e2b00a admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:30.592873 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5fee240-4c73-4a57-9160-348fc4e2b00a admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:30.594193 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5fee240-4c73-4a57-9160-348fc4e2b00a admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:30.599826 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5fee240-4c73-4a57-9160-348fc4e2b00a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:30.603952 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5fee240-4c73-4a57-9160-348fc4e2b00a admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:30.623291 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 797/1593] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:33:30 2022] GET /identity/v3/roles => generated 1420 bytes in 75 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:33:30.625373 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5fee240-4c73-4a57-9160-348fc4e2b00a admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:30.641388 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5fee240-4c73-4a57-9160-348fc4e2b00a admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:30.651974 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d1f8e9e-9fe0-402c-8596-44f54853d90d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:30.653061 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d1f8e9e-9fe0-402c-8596-44f54853d90d admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:30.654101 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d1f8e9e-9fe0-402c-8596-44f54853d90d admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:30.654101 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d1f8e9e-9fe0-402c-8596-44f54853d90d admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:30.655417 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d1f8e9e-9fe0-402c-8596-44f54853d90d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:30.658093 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d1f8e9e-9fe0-402c-8596-44f54853d90d admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:30.660797 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb551f7a-576d-439f-be69-0a0b0a77a400 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:30.671738 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16b17995-86ac-46f7-a801-c9a0159fafe4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:30.672854 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b17995-86ac-46f7-a801-c9a0159fafe4 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:30.672854 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b17995-86ac-46f7-a801-c9a0159fafe4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:30.673205 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b17995-86ac-46f7-a801-c9a0159fafe4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:30.679568 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 798/1594] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:33:30 2022] GET /identity/v3/roles => generated 1420 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:33:30.682876 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d1f8e9e-9fe0-402c-8596-44f54853d90d admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:30.701854 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d1f8e9e-9fe0-402c-8596-44f54853d90d admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:30.703272 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16b17995-86ac-46f7-a801-c9a0159fafe4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:30.704991 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16b17995-86ac-46f7-a801-c9a0159fafe4 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:30.712746 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-239a3955-c15a-4b08-b8b5-e8eb23930b7a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:30.714187 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-239a3955-c15a-4b08-b8b5-e8eb23930b7a admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:30.714473 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-239a3955-c15a-4b08-b8b5-e8eb23930b7a admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:30.714827 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-239a3955-c15a-4b08-b8b5-e8eb23930b7a admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:30.715515 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-239a3955-c15a-4b08-b8b5-e8eb23930b7a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:30.717951 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-239a3955-c15a-4b08-b8b5-e8eb23930b7a admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:30.735004 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 799/1595] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:33:30 2022] GET /identity/v3/roles => generated 1420 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:33:30.736007 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-239a3955-c15a-4b08-b8b5-e8eb23930b7a admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:30.744335 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 797/1596] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:33:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 202 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:33:30.749976 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16b17995-86ac-46f7-a801-c9a0159fafe4 service neutron] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:30.751444 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-239a3955-c15a-4b08-b8b5-e8eb23930b7a admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:30.759224 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0caea49d-7e76-4a3e-82e4-811c127ec4f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:30.759891 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0caea49d-7e76-4a3e-82e4-811c127ec4f6 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:30.760179 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0caea49d-7e76-4a3e-82e4-811c127ec4f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:30.760497 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0caea49d-7e76-4a3e-82e4-811c127ec4f6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:30.761215 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0caea49d-7e76-4a3e-82e4-811c127ec4f6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:30.763172 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0caea49d-7e76-4a3e-82e4-811c127ec4f6 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:30.781681 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 800/1597] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:33:30 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:33:30.781681 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0caea49d-7e76-4a3e-82e4-811c127ec4f6 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:30.794335 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16b17995-86ac-46f7-a801-c9a0159fafe4 service neutron] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:30.807105 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c5cbbbd-9c54-4557-ae47-0eba3017ee5e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:30.808690 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c5cbbbd-9c54-4557-ae47-0eba3017ee5e admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:30.809014 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c5cbbbd-9c54-4557-ae47-0eba3017ee5e admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:30.809679 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c5cbbbd-9c54-4557-ae47-0eba3017ee5e admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:30.810694 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c5cbbbd-9c54-4557-ae47-0eba3017ee5e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:30.812678 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c5cbbbd-9c54-4557-ae47-0eba3017ee5e admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:30.817994 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:33:30.818200 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0caea49d-7e76-4a3e-82e4-811c127ec4f6 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:30.826379 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fded33a0-d676-47d4-ad74-7cf73e2109f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:30.827523 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fded33a0-d676-47d4-ad74-7cf73e2109f9 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:30.828067 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fded33a0-d676-47d4-ad74-7cf73e2109f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:30.828550 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fded33a0-d676-47d4-ad74-7cf73e2109f9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:30.833722 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fded33a0-d676-47d4-ad74-7cf73e2109f9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:30.835605 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fded33a0-d676-47d4-ad74-7cf73e2109f9 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:30.847886 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 798/1598] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:33:30 2022] GET /identity/v3/roles => generated 1420 bytes in 103 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:33:30.849334 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c5cbbbd-9c54-4557-ae47-0eba3017ee5e admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:30.865235 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 801/1599] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:33:30 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:33:30.866605 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fded33a0-d676-47d4-ad74-7cf73e2109f9 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:30.883053 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fded33a0-d676-47d4-ad74-7cf73e2109f9 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:30.886797 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c5cbbbd-9c54-4557-ae47-0eba3017ee5e admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:30.891976 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbc4d125-47d2-4fb5-b247-2cda6b01f219 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:30.893338 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbc4d125-47d2-4fb5-b247-2cda6b01f219 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:30.894356 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbc4d125-47d2-4fb5-b247-2cda6b01f219 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:30.895004 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbc4d125-47d2-4fb5-b247-2cda6b01f219 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:30.895920 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbc4d125-47d2-4fb5-b247-2cda6b01f219 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:30.898393 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbc4d125-47d2-4fb5-b247-2cda6b01f219 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:30.906693 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e498c96-5b4d-4b09-a1b4-e4b0afc65091 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:30.907764 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e498c96-5b4d-4b09-a1b4-e4b0afc65091 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:30.908173 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e498c96-5b4d-4b09-a1b4-e4b0afc65091 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:30.908569 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e498c96-5b4d-4b09-a1b4-e4b0afc65091 admin admin] PATH_INFO: `/v3/projects/d3c829bce41c4be9a0027076ef0ace75/users/bbbfb39899b04668ac1b7ff28d18e839/roles/02e242d5520740d4b77a5f4008c7eb57` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:30.922410 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 802/1600] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:33:30 2022] GET /identity/v3/roles => generated 1420 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:33:30.923733 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbc4d125-47d2-4fb5-b247-2cda6b01f219 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:30.943850 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbc4d125-47d2-4fb5-b247-2cda6b01f219 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:30.947084 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e498c96-5b4d-4b09-a1b4-e4b0afc65091 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d3c829bce41c4be9a0027076ef0ace75, user_id=bbbfb39899b04668ac1b7ff28d18e839, role_id=02e242d5520740d4b77a5f4008c7eb57)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:30.956287 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-931aabe6-b12f-4f97-9f0b-0af8f2777915 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:30.957064 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e498c96-5b4d-4b09-a1b4-e4b0afc65091 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:30.958241 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-931aabe6-b12f-4f97-9f0b-0af8f2777915 admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:30.958937 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-931aabe6-b12f-4f97-9f0b-0af8f2777915 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:30.959681 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-931aabe6-b12f-4f97-9f0b-0af8f2777915 admin admin] PATH_INFO: `/v3/projects/1973a2c77684405281681124fcf2a09c/users/691ad2f5ff824b9490cd18024edfe508/roles/02e242d5520740d4b77a5f4008c7eb57` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:30.991120 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 799/1601] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:33:30 2022] PUT /identity/v3/projects/d3c829bce41c4be9a0027076ef0ace75/users/bbbfb39899b04668ac1b7ff28d18e839/roles/02e242d5520740d4b77a5f4008c7eb57 => generated 0 bytes in 143 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:33:30.992195 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e498c96-5b4d-4b09-a1b4-e4b0afc65091 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:31.000197 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-931aabe6-b12f-4f97-9f0b-0af8f2777915 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1973a2c77684405281681124fcf2a09c, user_id=691ad2f5ff824b9490cd18024edfe508, role_id=02e242d5520740d4b77a5f4008c7eb57)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:31.002964 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: WARNING py.warnings [None req-931aabe6-b12f-4f97-9f0b-0af8f2777915 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 29 21:33:31.002964 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: warnings.warn(msg) Aug 29 21:33:31.002964 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: Aug 29 21:33:31.006200 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-931aabe6-b12f-4f97-9f0b-0af8f2777915 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:31.034552 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 803/1602] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:33:30 2022] PUT /identity/v3/projects/1973a2c77684405281681124fcf2a09c/users/691ad2f5ff824b9490cd18024edfe508/roles/02e242d5520740d4b77a5f4008c7eb57 => generated 0 bytes in 111 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:33:31.037623 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-931aabe6-b12f-4f97-9f0b-0af8f2777915 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:31.212511 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e498c96-5b4d-4b09-a1b4-e4b0afc65091 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:31.236369 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bd1d2c9-9731-4dea-8d83-ea986b965efb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:31.237903 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-931aabe6-b12f-4f97-9f0b-0af8f2777915 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:31.238106 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bd1d2c9-9731-4dea-8d83-ea986b965efb admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:31.239531 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bd1d2c9-9731-4dea-8d83-ea986b965efb admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:31.239934 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bd1d2c9-9731-4dea-8d83-ea986b965efb admin admin] PATH_INFO: `/v3/projects/26439244d8a242598ddc188d87b97961/users/b4598ca6321e4a9786a03b5b541f7232/roles/02e242d5520740d4b77a5f4008c7eb57` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:31.254210 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-347d09ac-fcaf-4dec-8299-7decea9138fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:31.255751 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-347d09ac-fcaf-4dec-8299-7decea9138fe admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:31.256510 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-347d09ac-fcaf-4dec-8299-7decea9138fe admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:31.257531 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-347d09ac-fcaf-4dec-8299-7decea9138fe admin admin] PATH_INFO: `/v3/projects/cbaa820ba6ab47108ffc321f1011c3e6/users/2eabcdd8526f488291dbb237018c29ae/roles/02e242d5520740d4b77a5f4008c7eb57` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:31.304462 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-347d09ac-fcaf-4dec-8299-7decea9138fe admin admin] RBAC: Authorizing `identity:create_grant(project_id=cbaa820ba6ab47108ffc321f1011c3e6, user_id=2eabcdd8526f488291dbb237018c29ae, role_id=02e242d5520740d4b77a5f4008c7eb57)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:31.308024 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bd1d2c9-9731-4dea-8d83-ea986b965efb admin admin] RBAC: Authorizing `identity:create_grant(project_id=26439244d8a242598ddc188d87b97961, user_id=b4598ca6321e4a9786a03b5b541f7232, role_id=02e242d5520740d4b77a5f4008c7eb57)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:31.315408 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-347d09ac-fcaf-4dec-8299-7decea9138fe admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:31.318996 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bd1d2c9-9731-4dea-8d83-ea986b965efb admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:31.357926 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 800/1603] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:33:30 2022] PUT /identity/v3/projects/26439244d8a242598ddc188d87b97961/users/b4598ca6321e4a9786a03b5b541f7232/roles/02e242d5520740d4b77a5f4008c7eb57 => generated 0 bytes in 366 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:33:31.359756 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bd1d2c9-9731-4dea-8d83-ea986b965efb admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:31.368091 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 804/1604] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:33:31 2022] PUT /identity/v3/projects/cbaa820ba6ab47108ffc321f1011c3e6/users/2eabcdd8526f488291dbb237018c29ae/roles/02e242d5520740d4b77a5f4008c7eb57 => generated 0 bytes in 332 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:33:31.370458 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-347d09ac-fcaf-4dec-8299-7decea9138fe admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:31.496071 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bd1d2c9-9731-4dea-8d83-ea986b965efb admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:31.535125 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:33:31.566241 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-347d09ac-fcaf-4dec-8299-7decea9138fe admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:31.582155 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86bd48ca-c2d2-4c38-bf7b-830a34cf98d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:31.586794 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86bd48ca-c2d2-4c38-bf7b-830a34cf98d5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:31.587565 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86bd48ca-c2d2-4c38-bf7b-830a34cf98d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:31.588081 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86bd48ca-c2d2-4c38-bf7b-830a34cf98d5 admin admin] PATH_INFO: `/v3/projects/71f4912b8f1b4a5199fe6d11315c2658/users/96d3a0e28dc247b5a7ee9434a4d6554c/roles/02e242d5520740d4b77a5f4008c7eb57` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:31.612352 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86bd48ca-c2d2-4c38-bf7b-830a34cf98d5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=71f4912b8f1b4a5199fe6d11315c2658, user_id=96d3a0e28dc247b5a7ee9434a4d6554c, role_id=02e242d5520740d4b77a5f4008c7eb57)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:31.617295 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86bd48ca-c2d2-4c38-bf7b-830a34cf98d5 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:31.630334 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-085b0c41-a0df-4f2c-ac85-ec0c51f04469 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:31.632540 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-085b0c41-a0df-4f2c-ac85-ec0c51f04469 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:31.633861 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-085b0c41-a0df-4f2c-ac85-ec0c51f04469 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:31.634571 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-085b0c41-a0df-4f2c-ac85-ec0c51f04469 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:31.637568 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-085b0c41-a0df-4f2c-ac85-ec0c51f04469 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:31.640398 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-085b0c41-a0df-4f2c-ac85-ec0c51f04469 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:31.651165 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 805/1605] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:33:31 2022] PUT /identity/v3/projects/71f4912b8f1b4a5199fe6d11315c2658/users/96d3a0e28dc247b5a7ee9434a4d6554c/roles/02e242d5520740d4b77a5f4008c7eb57 => generated 0 bytes in 277 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:33:31.651165 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cab665e6-8f33-467d-be0b-6fa0d48417a2 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:31.651165 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cab665e6-8f33-467d-be0b-6fa0d48417a2 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:31.651165 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cab665e6-8f33-467d-be0b-6fa0d48417a2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:31.657488 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:33:31.667932 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 801/1606] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:33:31 2022] GET /identity/v3/roles => generated 1420 bytes in 309 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:33:31.671110 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-325379fa-aea3-4cf0-9c31-0d572b932ef7 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:31.671885 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-325379fa-aea3-4cf0-9c31-0d572b932ef7 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:31.674262 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-325379fa-aea3-4cf0-9c31-0d572b932ef7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:31.701313 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-cab665e6-8f33-467d-be0b-6fa0d48417a2 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:33:31.707352 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-cab665e6-8f33-467d-be0b-6fa0d48417a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:33:31.729708 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-325379fa-aea3-4cf0-9c31-0d572b932ef7 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:33:31.740943 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-325379fa-aea3-4cf0-9c31-0d572b932ef7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:33:31.882071 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 806/1607] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:33:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 235 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:33:31.886063 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab8784f8-0547-4aec-9ce4-c9dbd8c3b69b None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:31.886670 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab8784f8-0547-4aec-9ce4-c9dbd8c3b69b None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:31.887644 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab8784f8-0547-4aec-9ce4-c9dbd8c3b69b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:31.935857 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 802/1608] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:33:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 267 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:33:31.946487 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d63e983f-778a-498c-967c-ca910f61fd3a None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:31.946487 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d63e983f-778a-498c-967c-ca910f61fd3a None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:31.946487 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d63e983f-778a-498c-967c-ca910f61fd3a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:31.958775 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-ab8784f8-0547-4aec-9ce4-c9dbd8c3b69b None None] MFA Rules not processed for user `b4598ca6321e4a9786a03b5b541f7232`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:33:31.959766 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-ab8784f8-0547-4aec-9ce4-c9dbd8c3b69b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:33:31.991463 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-d63e983f-778a-498c-967c-ca910f61fd3a None None] MFA Rules not processed for user `2eabcdd8526f488291dbb237018c29ae`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:33:31.994212 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-d63e983f-778a-498c-967c-ca910f61fd3a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:33:32.094745 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 807/1609] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:33:31 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 208 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:33:32.094745 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb12364e-4186-44b1-b32b-b8d71767a26f None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:32.094745 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb12364e-4186-44b1-b32b-b8d71767a26f None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:32.094745 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb12364e-4186-44b1-b32b-b8d71767a26f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:32.203896 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 803/1610] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:33:31 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 228 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:33:32.203896 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d63e983f-778a-498c-967c-ca910f61fd3a None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:32.204821 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-bb12364e-4186-44b1-b32b-b8d71767a26f None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:33:32.204821 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-bb12364e-4186-44b1-b32b-b8d71767a26f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:33:32.205133 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d63e983f-778a-498c-967c-ca910f61fd3a None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:32.205133 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8b2e9df-bc63-46ea-9abf-e762cf3f663d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:32.205133 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8b2e9df-bc63-46ea-9abf-e762cf3f663d admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:32.205133 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8b2e9df-bc63-46ea-9abf-e762cf3f663d admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:32.205453 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8b2e9df-bc63-46ea-9abf-e762cf3f663d admin admin] PATH_INFO: `/v3/projects/3057e78899314a99bf88c55ab9b0fa25/users/c2f53bb29a90436782679aed93fa2c49/roles/02e242d5520740d4b77a5f4008c7eb57` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:32.212438 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 808/1611] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:33:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:33:32.213915 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb12364e-4186-44b1-b32b-b8d71767a26f None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:32.240726 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8b2e9df-bc63-46ea-9abf-e762cf3f663d admin admin] RBAC: Authorizing `identity:create_grant(project_id=3057e78899314a99bf88c55ab9b0fa25, user_id=c2f53bb29a90436782679aed93fa2c49, role_id=02e242d5520740d4b77a5f4008c7eb57)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:32.247068 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8b2e9df-bc63-46ea-9abf-e762cf3f663d admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:32.286677 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 804/1612] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:33:32 2022] PUT /identity/v3/projects/3057e78899314a99bf88c55ab9b0fa25/users/c2f53bb29a90436782679aed93fa2c49/roles/02e242d5520740d4b77a5f4008c7eb57 => generated 0 bytes in 122 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:33:32.288761 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8b2e9df-bc63-46ea-9abf-e762cf3f663d admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:32.387287 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb12364e-4186-44b1-b32b-b8d71767a26f None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:32.413107 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:33:32.458469 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8b2e9df-bc63-46ea-9abf-e762cf3f663d admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:32.465728 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42f628b5-bb58-4e51-80ce-d989f8b27860 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:32.466987 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42f628b5-bb58-4e51-80ce-d989f8b27860 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:32.467613 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42f628b5-bb58-4e51-80ce-d989f8b27860 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:32.467926 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42f628b5-bb58-4e51-80ce-d989f8b27860 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:32.482490 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-072b4a04-48a5-4da1-bcbd-3114f81ea685 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:32.485509 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-072b4a04-48a5-4da1-bcbd-3114f81ea685 service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:32.486315 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-072b4a04-48a5-4da1-bcbd-3114f81ea685 service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:32.490430 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-072b4a04-48a5-4da1-bcbd-3114f81ea685 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:32.509413 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:33:32.574862 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42f628b5-bb58-4e51-80ce-d989f8b27860 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:32.577709 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42f628b5-bb58-4e51-80ce-d989f8b27860 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:32.599172 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 805/1613] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:33:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:33:32.600783 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42f628b5-bb58-4e51-80ce-d989f8b27860 service neutron] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:32.616660 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-072b4a04-48a5-4da1-bcbd-3114f81ea685 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:32.620000 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-072b4a04-48a5-4da1-bcbd-3114f81ea685 service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:32.629234 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42f628b5-bb58-4e51-80ce-d989f8b27860 service neutron] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:32.636298 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 809/1614] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:33:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 423 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:33:32.637628 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd2e5706-6e62-447e-89d6-00e014809513 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:32.638216 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-072b4a04-48a5-4da1-bcbd-3114f81ea685 service neutron] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:32.638583 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd2e5706-6e62-447e-89d6-00e014809513 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:32.639133 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd2e5706-6e62-447e-89d6-00e014809513 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:32.639453 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd2e5706-6e62-447e-89d6-00e014809513 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:32.640377 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd2e5706-6e62-447e-89d6-00e014809513 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:32.644088 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd2e5706-6e62-447e-89d6-00e014809513 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:32.694180 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 806/1615] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:33:32 2022] POST /identity/v3/projects => generated 369 bytes in 90 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:33:32.694180 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd2e5706-6e62-447e-89d6-00e014809513 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:32.752982 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-072b4a04-48a5-4da1-bcbd-3114f81ea685 service neutron] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:32.837457 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd2e5706-6e62-447e-89d6-00e014809513 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:32.875775 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92037be1-809b-4bd2-864c-4121675f81d0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:32.876883 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92037be1-809b-4bd2-864c-4121675f81d0 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:32.877502 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92037be1-809b-4bd2-864c-4121675f81d0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:32.878151 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92037be1-809b-4bd2-864c-4121675f81d0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:32.905282 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-874d794d-7a9c-49f0-a56d-f380c2b0febd service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dc226f7ddcb49ecacb51ce57fdfc11b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:32.907805 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-874d794d-7a9c-49f0-a56d-f380c2b0febd service cinder] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:32.908908 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-874d794d-7a9c-49f0-a56d-f380c2b0febd service cinder] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:32.910373 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-874d794d-7a9c-49f0-a56d-f380c2b0febd service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:33.060915 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:33:33.093298 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92037be1-809b-4bd2-864c-4121675f81d0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:33.098841 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92037be1-809b-4bd2-864c-4121675f81d0 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:33.123396 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-874d794d-7a9c-49f0-a56d-f380c2b0febd service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:33.126406 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-874d794d-7a9c-49f0-a56d-f380c2b0febd service cinder] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:33.136311 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 807/1616] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:33:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 446 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:33:33.139795 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-600509e0-9181-4a83-b5bf-887eaec8930b None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:33.139795 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-600509e0-9181-4a83-b5bf-887eaec8930b None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:33.139795 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-600509e0-9181-4a83-b5bf-887eaec8930b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:33.172104 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 810/1617] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:33:32 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 535 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:33:33.173560 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-874d794d-7a9c-49f0-a56d-f380c2b0febd service cinder] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:33.215159 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-874d794d-7a9c-49f0-a56d-f380c2b0febd service cinder] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:33.217940 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-600509e0-9181-4a83-b5bf-887eaec8930b None None] MFA Rules not processed for user `c2f53bb29a90436782679aed93fa2c49`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:33:33.219371 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-600509e0-9181-4a83-b5bf-887eaec8930b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:33:33.241623 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6253d154-9eac-4f8e-addb-0a1aae6a4afe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:33.242721 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6253d154-9eac-4f8e-addb-0a1aae6a4afe admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:33.243238 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6253d154-9eac-4f8e-addb-0a1aae6a4afe admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:33.243730 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6253d154-9eac-4f8e-addb-0a1aae6a4afe admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:33.244886 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6253d154-9eac-4f8e-addb-0a1aae6a4afe admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:33.248726 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6253d154-9eac-4f8e-addb-0a1aae6a4afe admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:33.317253 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:33:33.322272 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 811/1618] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:33:33 2022] POST /identity/v3/users => generated 291 bytes in 149 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:33:33.328457 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6253d154-9eac-4f8e-addb-0a1aae6a4afe admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:33.369020 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6253d154-9eac-4f8e-addb-0a1aae6a4afe admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:33.391154 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8e2628d-90fd-4a51-aec5-dbc6c1010cf1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:33.392229 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8e2628d-90fd-4a51-aec5-dbc6c1010cf1 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:33.392717 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8e2628d-90fd-4a51-aec5-dbc6c1010cf1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:33.394089 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8e2628d-90fd-4a51-aec5-dbc6c1010cf1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:33.399526 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8e2628d-90fd-4a51-aec5-dbc6c1010cf1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:33.406358 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8e2628d-90fd-4a51-aec5-dbc6c1010cf1 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:33.416126 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 808/1619] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:33:33 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 279 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:33:33.464915 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 812/1620] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:33:33 2022] GET /identity/v3/roles => generated 1420 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:33:33.472091 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-600509e0-9181-4a83-b5bf-887eaec8930b None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:33.479295 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c6bfbc7-2b76-4953-88df-65fa24b7cd93 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:33.479769 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c6bfbc7-2b76-4953-88df-65fa24b7cd93 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:33.480119 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c6bfbc7-2b76-4953-88df-65fa24b7cd93 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:33.481978 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 813/1621] 10.222.0.90 () {54 vars in 962 bytes} [Mon Aug 29 21:33:33 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:33:33.513551 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-600509e0-9181-4a83-b5bf-887eaec8930b None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:33.516970 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6966f4a4-f58b-4f0b-a8e3-f32fc95aa412 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:33.517476 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6966f4a4-f58b-4f0b-a8e3-f32fc95aa412 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:33.517957 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6966f4a4-f58b-4f0b-a8e3-f32fc95aa412 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:33.533656 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68308484-c5a2-458d-9fc9-c75121e66e6f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:33.534580 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68308484-c5a2-458d-9fc9-c75121e66e6f admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:33.535011 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68308484-c5a2-458d-9fc9-c75121e66e6f admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:33.535778 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68308484-c5a2-458d-9fc9-c75121e66e6f admin admin] PATH_INFO: `/v3/projects/d686968826f340cfac2cf7bdb1bb8d58/users/8bdb7810412d409c9e8aa202ceb6463c/roles/a6e562cdc96c4be691d638990f8403fd` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:33.578661 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68308484-c5a2-458d-9fc9-c75121e66e6f admin admin] RBAC: Authorizing `identity:create_grant(project_id=d686968826f340cfac2cf7bdb1bb8d58, user_id=8bdb7810412d409c9e8aa202ceb6463c, role_id=a6e562cdc96c4be691d638990f8403fd)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:33.587788 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68308484-c5a2-458d-9fc9-c75121e66e6f admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:33.629880 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 809/1622] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:33:33 2022] PUT /identity/v3/projects/d686968826f340cfac2cf7bdb1bb8d58/users/8bdb7810412d409c9e8aa202ceb6463c/roles/a6e562cdc96c4be691d638990f8403fd => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 21:33:33.636843 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68308484-c5a2-458d-9fc9-c75121e66e6f admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:33.656829 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-6966f4a4-f58b-4f0b-a8e3-f32fc95aa412 None None] MFA Rules not processed for user `a35b5b7dea7b421da056f063d2436ffb`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:33:33.658555 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-6966f4a4-f58b-4f0b-a8e3-f32fc95aa412 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:33:33.853042 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68308484-c5a2-458d-9fc9-c75121e66e6f admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:33.875497 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 814/1623] 10.222.0.90 () {58 vars in 1059 bytes} [Mon Aug 29 21:33:33 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 360 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 21:33:33.890459 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6966f4a4-f58b-4f0b-a8e3-f32fc95aa412 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:33.892010 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae637758-1498-4f08-b309-5b7ae49fd3ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:33.892958 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae637758-1498-4f08-b309-5b7ae49fd3ce admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:33.893253 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae637758-1498-4f08-b309-5b7ae49fd3ce admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:33.893539 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae637758-1498-4f08-b309-5b7ae49fd3ce admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:33.894280 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae637758-1498-4f08-b309-5b7ae49fd3ce admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:33.897728 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae637758-1498-4f08-b309-5b7ae49fd3ce admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:33.913586 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6966f4a4-f58b-4f0b-a8e3-f32fc95aa412 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:33.924091 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 810/1624] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:33:33 2022] GET /identity/v3/roles => generated 1420 bytes in 283 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:33:33.926520 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae637758-1498-4f08-b309-5b7ae49fd3ce admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:33.945611 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d7d936e-bdb7-44ea-ae1d-cd024e687338 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a35b5b7dea7b421da056f063d2436ffb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:33.948084 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d7d936e-bdb7-44ea-ae1d-cd024e687338 service nova] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:33.950644 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d7d936e-bdb7-44ea-ae1d-cd024e687338 service nova] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:33.950917 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d7d936e-bdb7-44ea-ae1d-cd024e687338 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:33.953715 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae637758-1498-4f08-b309-5b7ae49fd3ce admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:33.981504 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7977455-5654-40f8-9ea0-98a79f4a0508 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:33.982810 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7977455-5654-40f8-9ea0-98a79f4a0508 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:33.983103 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7977455-5654-40f8-9ea0-98a79f4a0508 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:33.983387 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7977455-5654-40f8-9ea0-98a79f4a0508 admin admin] PATH_INFO: `/v3/projects/d686968826f340cfac2cf7bdb1bb8d58/users/8bdb7810412d409c9e8aa202ceb6463c/roles/a6e562cdc96c4be691d638990f8403fd` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:33.992047 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7977455-5654-40f8-9ea0-98a79f4a0508 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d686968826f340cfac2cf7bdb1bb8d58, user_id=8bdb7810412d409c9e8aa202ceb6463c, role_id=a6e562cdc96c4be691d638990f8403fd)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:34.006715 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7977455-5654-40f8-9ea0-98a79f4a0508 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:34.049885 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 811/1625] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:33:33 2022] PUT /identity/v3/projects/d686968826f340cfac2cf7bdb1bb8d58/users/8bdb7810412d409c9e8aa202ceb6463c/roles/a6e562cdc96c4be691d638990f8403fd => generated 0 bytes in 122 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 21:33:34.069628 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffa485c4-9cd9-4d07-ac15-80dbe3ecd0d4 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:34.071054 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffa485c4-9cd9-4d07-ac15-80dbe3ecd0d4 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:34.073385 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffa485c4-9cd9-4d07-ac15-80dbe3ecd0d4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:34.149993 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d7d936e-bdb7-44ea-ae1d-cd024e687338 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:34.149993 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d7d936e-bdb7-44ea-ae1d-cd024e687338 service nova] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:34.186511 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 815/1626] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:33:33 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 297 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 21:33:34.235878 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-ffa485c4-9cd9-4d07-ac15-80dbe3ecd0d4 None None] MFA Rules not processed for user `8bdb7810412d409c9e8aa202ceb6463c`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:33:34.242106 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-ffa485c4-9cd9-4d07-ac15-80dbe3ecd0d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:33:34.554504 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 812/1627] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:33:34 2022] POST /identity/v3/auth/tokens => generated 3313 bytes in 488 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 21:33:34.568297 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d7d936e-bdb7-44ea-ae1d-cd024e687338 service nova] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:34.647750 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:33:34.755367 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d7d936e-bdb7-44ea-ae1d-cd024e687338 service nova] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:35.036423 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d8590b7-a48c-496d-9b95-7b8143f25eb6 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:35.037955 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d8590b7-a48c-496d-9b95-7b8143f25eb6 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:35.038680 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d8590b7-a48c-496d-9b95-7b8143f25eb6 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:35.042460 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 813/1628] 10.222.0.90 () {54 vars in 912 bytes} [Mon Aug 29 21:33:35 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 29 21:33:35.053116 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daf8948f-212f-42ff-8a2b-b76f55e97010 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:35.054167 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daf8948f-212f-42ff-8a2b-b76f55e97010 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:35.055562 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daf8948f-212f-42ff-8a2b-b76f55e97010 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:35.089532 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-582b123f-5433-4e1f-ba69-524fb91f8fc8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dc226f7ddcb49ecacb51ce57fdfc11b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:35.094495 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-582b123f-5433-4e1f-ba69-524fb91f8fc8 service cinder] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:35.095023 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-582b123f-5433-4e1f-ba69-524fb91f8fc8 service cinder] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:35.096316 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-582b123f-5433-4e1f-ba69-524fb91f8fc8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:35.129907 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-daf8948f-212f-42ff-8a2b-b76f55e97010 None None] MFA Rules not processed for user `a35b5b7dea7b421da056f063d2436ffb`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:33:35.132098 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-daf8948f-212f-42ff-8a2b-b76f55e97010 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:33:35.135195 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-582b123f-5433-4e1f-ba69-524fb91f8fc8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:35.141976 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-582b123f-5433-4e1f-ba69-524fb91f8fc8 service cinder] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:35.193737 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:33:35.213833 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 816/1629] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:33:34 2022] GET /identity/v3/auth/tokens => generated 3313 bytes in 646 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:33:35.388506 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 814/1630] 10.222.0.90 () {58 vars in 1009 bytes} [Mon Aug 29 21:33:35 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 337 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 21:33:35.418903 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-582b123f-5433-4e1f-ba69-524fb91f8fc8 service cinder] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:35.451956 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-582b123f-5433-4e1f-ba69-524fb91f8fc8 service cinder] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:35.468044 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b1702a2-ad3b-4179-b799-e7e613b7f671 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dc226f7ddcb49ecacb51ce57fdfc11b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:35.468706 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b1702a2-ad3b-4179-b799-e7e613b7f671 service cinder] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:35.468989 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b1702a2-ad3b-4179-b799-e7e613b7f671 service cinder] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:35.469265 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b1702a2-ad3b-4179-b799-e7e613b7f671 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:35.509127 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b1702a2-ad3b-4179-b799-e7e613b7f671 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:35.522955 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b1702a2-ad3b-4179-b799-e7e613b7f671 service cinder] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:35.569812 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 817/1631] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:33:35 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 152 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:33:36.927045 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efb061b7-3867-4a77-aecf-13c73665b7ee None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:36.927727 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efb061b7-3867-4a77-aecf-13c73665b7ee None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:36.927947 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efb061b7-3867-4a77-aecf-13c73665b7ee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:37.075514 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-efb061b7-3867-4a77-aecf-13c73665b7ee None None] MFA Rules not processed for user `691ad2f5ff824b9490cd18024edfe508`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:33:37.076822 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-efb061b7-3867-4a77-aecf-13c73665b7ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:33:37.393953 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 815/1632] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:33:36 2022] POST /identity/v3/auth/tokens => generated 3280 bytes in 467 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:33:37.401269 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b1702a2-ad3b-4179-b799-e7e613b7f671 service cinder] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:37.502801 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c9960aa-a039-478c-a7fb-8b84cfa7185a None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:37.503160 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c9960aa-a039-478c-a7fb-8b84cfa7185a None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:37.503462 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c9960aa-a039-478c-a7fb-8b84cfa7185a None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:37.506042 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 816/1633] 10.222.0.90 () {54 vars in 962 bytes} [Mon Aug 29 21:33:37 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 29 21:33:37.533801 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86626cb4-8bc5-4df5-9c47-455563a9e2dc None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:37.534143 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86626cb4-8bc5-4df5-9c47-455563a9e2dc None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:37.534476 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86626cb4-8bc5-4df5-9c47-455563a9e2dc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:37.599724 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-86626cb4-8bc5-4df5-9c47-455563a9e2dc None None] MFA Rules not processed for user `a35b5b7dea7b421da056f063d2436ffb`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:33:37.601324 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-86626cb4-8bc5-4df5-9c47-455563a9e2dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:33:37.635688 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b1702a2-ad3b-4179-b799-e7e613b7f671 service cinder] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:37.645630 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 817/1634] 10.222.0.90 () {58 vars in 1059 bytes} [Mon Aug 29 21:33:37 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 21:33:37.685239 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bce2b326-0de1-4642-ba75-0e7930dbc395 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:37.685333 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72fe4887-a852-4f56-93b5-6c81d6cb537e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:37.685419 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bce2b326-0de1-4642-ba75-0e7930dbc395 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:37.685419 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bce2b326-0de1-4642-ba75-0e7930dbc395 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:37.685551 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72fe4887-a852-4f56-93b5-6c81d6cb537e admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:37.685551 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72fe4887-a852-4f56-93b5-6c81d6cb537e admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:37.685551 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72fe4887-a852-4f56-93b5-6c81d6cb537e admin admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:37.685551 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72fe4887-a852-4f56-93b5-6c81d6cb537e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:37.685814 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 818/1635] 10.222.0.90 () {54 vars in 912 bytes} [Mon Aug 29 21:33:37 2022] GET /identity => generated 270 bytes in 26 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:33:37.685905 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72fe4887-a852-4f56-93b5-6c81d6cb537e admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:37.700578 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92472c8f-8ce9-48e1-8644-43a41501831f None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:37.703055 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92472c8f-8ce9-48e1-8644-43a41501831f None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:37.703439 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92472c8f-8ce9-48e1-8644-43a41501831f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:37.760695 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-92472c8f-8ce9-48e1-8644-43a41501831f None None] MFA Rules not processed for user `a35b5b7dea7b421da056f063d2436ffb`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:33:37.760695 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-92472c8f-8ce9-48e1-8644-43a41501831f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:33:37.802544 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 818/1636] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:33:37 2022] POST /identity/v3/projects => generated 405 bytes in 398 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 21:33:37.802927 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 819/1637] 10.222.0.90 () {58 vars in 1009 bytes} [Mon Aug 29 21:33:37 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 21:33:37.821227 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72fe4887-a852-4f56-93b5-6c81d6cb537e admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:37.843313 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92472c8f-8ce9-48e1-8644-43a41501831f None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:37.894148 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92472c8f-8ce9-48e1-8644-43a41501831f None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:37.977821 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:33:37.986355 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72fe4887-a852-4f56-93b5-6c81d6cb537e admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:38.003245 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03d572d5-8644-43a8-ac18-11b291c49efd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:38.004004 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03d572d5-8644-43a8-ac18-11b291c49efd admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:38.010110 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03d572d5-8644-43a8-ac18-11b291c49efd admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:38.010110 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03d572d5-8644-43a8-ac18-11b291c49efd admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:38.010110 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03d572d5-8644-43a8-ac18-11b291c49efd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:38.010110 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03d572d5-8644-43a8-ac18-11b291c49efd admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:38.084884 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 819/1638] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:33:37 2022] POST /identity/v3/users => generated 309 bytes in 283 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:33:38.094772 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03d572d5-8644-43a8-ac18-11b291c49efd admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:38.123764 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03d572d5-8644-43a8-ac18-11b291c49efd admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:38.139962 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6908f29-43fb-43b4-8858-01a35e94a1a6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dc226f7ddcb49ecacb51ce57fdfc11b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:38.141686 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6908f29-43fb-43b4-8858-01a35e94a1a6 service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:38.142257 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6908f29-43fb-43b4-8858-01a35e94a1a6 service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:38.142862 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6908f29-43fb-43b4-8858-01a35e94a1a6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:38.143794 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e410c6f0-3360-4911-9581-d52d2daaa947 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:38.144690 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e410c6f0-3360-4911-9581-d52d2daaa947 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:38.144982 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e410c6f0-3360-4911-9581-d52d2daaa947 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:38.145264 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e410c6f0-3360-4911-9581-d52d2daaa947 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:38.146021 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e410c6f0-3360-4911-9581-d52d2daaa947 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:38.147848 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e410c6f0-3360-4911-9581-d52d2daaa947 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:38.183829 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 820/1639] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:33:38 2022] GET /identity/v3/roles => generated 1420 bytes in 93 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 29 21:33:38.188523 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e410c6f0-3360-4911-9581-d52d2daaa947 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:38.226264 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e410c6f0-3360-4911-9581-d52d2daaa947 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:38.238677 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bed5c4a2-732e-4112-9336-d8a57b15b52a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:38.239396 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed5c4a2-732e-4112-9336-d8a57b15b52a admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:38.239683 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed5c4a2-732e-4112-9336-d8a57b15b52a admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:38.239959 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed5c4a2-732e-4112-9336-d8a57b15b52a admin admin] PATH_INFO: `/v3/projects/bce8b2975fab487c8bb797c80a8c7a4f/users/addde520faa7413b8f351e49d42dabb9/roles/a6e562cdc96c4be691d638990f8403fd` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:38.281287 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bed5c4a2-732e-4112-9336-d8a57b15b52a admin admin] RBAC: Authorizing `identity:create_grant(project_id=bce8b2975fab487c8bb797c80a8c7a4f, user_id=addde520faa7413b8f351e49d42dabb9, role_id=a6e562cdc96c4be691d638990f8403fd)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:38.285086 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bed5c4a2-732e-4112-9336-d8a57b15b52a admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:38.297144 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:33:38.314291 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 821/1640] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:33:38 2022] PUT /identity/v3/projects/bce8b2975fab487c8bb797c80a8c7a4f/users/addde520faa7413b8f351e49d42dabb9/roles/a6e562cdc96c4be691d638990f8403fd => generated 0 bytes in 126 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:33:38.319725 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bed5c4a2-732e-4112-9336-d8a57b15b52a admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:38.408524 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6908f29-43fb-43b4-8858-01a35e94a1a6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:38.410425 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6908f29-43fb-43b4-8858-01a35e94a1a6 service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:38.441643 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 820/1641] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:33:37 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 599 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:33:38.522232 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bed5c4a2-732e-4112-9336-d8a57b15b52a admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:38.553848 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90680234-8475-4f18-942c-89682fb1438d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:38.559231 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90680234-8475-4f18-942c-89682fb1438d admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:38.563098 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90680234-8475-4f18-942c-89682fb1438d admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:38.563445 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90680234-8475-4f18-942c-89682fb1438d admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:38.565201 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90680234-8475-4f18-942c-89682fb1438d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:38.577228 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90680234-8475-4f18-942c-89682fb1438d admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:38.616929 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 822/1642] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:33:38 2022] GET /identity/v3/roles => generated 1420 bytes in 298 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:33:38.621569 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6908f29-43fb-43b4-8858-01a35e94a1a6 service cinder] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:38.644935 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6908f29-43fb-43b4-8858-01a35e94a1a6 service cinder] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:38.651713 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ba6ed92-3d9a-4d19-aab1-0ed789ddbbfb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:38.652537 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ba6ed92-3d9a-4d19-aab1-0ed789ddbbfb admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:38.652901 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ba6ed92-3d9a-4d19-aab1-0ed789ddbbfb admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:38.653253 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ba6ed92-3d9a-4d19-aab1-0ed789ddbbfb admin admin] PATH_INFO: `/v3/projects/bce8b2975fab487c8bb797c80a8c7a4f/users/addde520faa7413b8f351e49d42dabb9/roles/a6e562cdc96c4be691d638990f8403fd` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:38.660499 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ba6ed92-3d9a-4d19-aab1-0ed789ddbbfb admin admin] RBAC: Authorizing `identity:create_grant(project_id=bce8b2975fab487c8bb797c80a8c7a4f, user_id=addde520faa7413b8f351e49d42dabb9, role_id=a6e562cdc96c4be691d638990f8403fd)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:38.669591 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ba6ed92-3d9a-4d19-aab1-0ed789ddbbfb admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:38.716131 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 821/1643] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:33:38 2022] PUT /identity/v3/projects/bce8b2975fab487c8bb797c80a8c7a4f/users/addde520faa7413b8f351e49d42dabb9/roles/a6e562cdc96c4be691d638990f8403fd => generated 0 bytes in 95 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:33:40.638714 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90680234-8475-4f18-942c-89682fb1438d admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:40.674399 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90680234-8475-4f18-942c-89682fb1438d admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:40.890900 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b590f702-6525-4efc-9116-b18554c53424 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:40.891807 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b590f702-6525-4efc-9116-b18554c53424 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:40.892212 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b590f702-6525-4efc-9116-b18554c53424 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:40.892607 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b590f702-6525-4efc-9116-b18554c53424 admin admin] PATH_INFO: `/v3/users/b4598ca6321e4a9786a03b5b541f7232` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:40.898116 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b590f702-6525-4efc-9116-b18554c53424 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b4598ca6321e4a9786a03b5b541f7232)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:40.900451 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: WARNING py.warnings [None req-b590f702-6525-4efc-9116-b18554c53424 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 29 21:33:40.900451 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: warnings.warn(msg) Aug 29 21:33:40.900451 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: Aug 29 21:33:40.902440 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b590f702-6525-4efc-9116-b18554c53424 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:41.158129 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-b590f702-6525-4efc-9116-b18554c53424 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b4598ca6321e4a9786a03b5b541f7232'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:33:41.160877 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=89,keystone:INSERT=4,keystone:DELETE=7 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:33:41.198492 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-b590f702-6525-4efc-9116-b18554c53424 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b4598ca6321e4a9786a03b5b541f7232'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:33:41.217272 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-b590f702-6525-4efc-9116-b18554c53424 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b4598ca6321e4a9786a03b5b541f7232'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:33:41.218049 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-b590f702-6525-4efc-9116-b18554c53424 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b4598ca6321e4a9786a03b5b541f7232'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:33:41.218686 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-b590f702-6525-4efc-9116-b18554c53424 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b4598ca6321e4a9786a03b5b541f7232'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:33:41.282658 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 823/1644] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:33:40 2022] DELETE /identity/v3/users/b4598ca6321e4a9786a03b5b541f7232 => generated 0 bytes in 641 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:33:41.284465 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5025b0a3-f9fe-474e-90fc-71e770800a47 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:41.285002 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5025b0a3-f9fe-474e-90fc-71e770800a47 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:41.285357 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5025b0a3-f9fe-474e-90fc-71e770800a47 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:41.365312 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-5025b0a3-f9fe-474e-90fc-71e770800a47 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:33:41.367885 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-5025b0a3-f9fe-474e-90fc-71e770800a47 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:33:41.553662 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 822/1645] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:33:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 267 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:33:41.645419 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b590f702-6525-4efc-9116-b18554c53424 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:41.648521 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-b590f702-6525-4efc-9116-b18554c53424 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:33:41.824178 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b590f702-6525-4efc-9116-b18554c53424 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:41.828994 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b418b4a-4884-424c-8222-8af03e00dd19 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:41.829360 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b418b4a-4884-424c-8222-8af03e00dd19 service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:41.829638 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b418b4a-4884-424c-8222-8af03e00dd19 service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:41.830006 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b418b4a-4884-424c-8222-8af03e00dd19 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:41.846837 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b418b4a-4884-424c-8222-8af03e00dd19 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:41.853888 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b418b4a-4884-424c-8222-8af03e00dd19 service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:41.875332 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 824/1646] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:33:41 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 231 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:33:42.021142 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41012394-20b0-4cd1-aa2b-7d8c3832c4b6 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:42.021636 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41012394-20b0-4cd1-aa2b-7d8c3832c4b6 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:42.022069 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41012394-20b0-4cd1-aa2b-7d8c3832c4b6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:42.074409 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-41012394-20b0-4cd1-aa2b-7d8c3832c4b6 None None] MFA Rules not processed for user `addde520faa7413b8f351e49d42dabb9`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:33:42.076725 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-41012394-20b0-4cd1-aa2b-7d8c3832c4b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:33:42.432893 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 823/1647] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:33:42 2022] POST /identity/v3/auth/tokens => generated 3349 bytes in 413 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:33:44.458523 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c54d76d9-75aa-4455-8fad-a362bbc762bd None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:44.460088 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c54d76d9-75aa-4455-8fad-a362bbc762bd None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:44.460738 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c54d76d9-75aa-4455-8fad-a362bbc762bd None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:44.481327 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 825/1648] 10.222.0.90 () {54 vars in 921 bytes} [Mon Aug 29 21:33:44 2022] GET /identity => generated 270 bytes in 21 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:33:44.494507 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41012394-20b0-4cd1-aa2b-7d8c3832c4b6 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:44.500784 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-41012394-20b0-4cd1-aa2b-7d8c3832c4b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:33:44.535912 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41012394-20b0-4cd1-aa2b-7d8c3832c4b6 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:44.557580 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fc46a15-d1cb-4d4c-af02-8888140971f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:44.560278 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc46a15-d1cb-4d4c-af02-8888140971f9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:44.567531 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc46a15-d1cb-4d4c-af02-8888140971f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:44.570393 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc46a15-d1cb-4d4c-af02-8888140971f9 admin admin] PATH_INFO: `/v3/projects/26439244d8a242598ddc188d87b97961` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:44.578610 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fc46a15-d1cb-4d4c-af02-8888140971f9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=26439244d8a242598ddc188d87b97961)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:44.582294 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: WARNING py.warnings [None req-5fc46a15-d1cb-4d4c-af02-8888140971f9 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 29 21:33:44.582294 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: warnings.warn(msg) Aug 29 21:33:44.582294 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: Aug 29 21:33:44.589289 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fc46a15-d1cb-4d4c-af02-8888140971f9 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:44.749538 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45c4424f-3c1a-468e-93a1-5f3ee79a04eb None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:44.749538 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45c4424f-3c1a-468e-93a1-5f3ee79a04eb None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:44.749538 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45c4424f-3c1a-468e-93a1-5f3ee79a04eb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:44.761209 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=97,keystone:INSERT=1,keystone:DELETE=2 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:33:44.827256 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-45c4424f-3c1a-468e-93a1-5f3ee79a04eb None None] MFA Rules not processed for user `b95fa07a7c0549dbaac49406e2639e21`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:33:44.830531 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-5fc46a15-d1cb-4d4c-af02-8888140971f9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '26439244d8a242598ddc188d87b97961', 'request_id': 'req-5fc46a15-d1cb-4d4c-af02-8888140971f9'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:33:44.832865 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-45c4424f-3c1a-468e-93a1-5f3ee79a04eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:33:44.893257 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-5fc46a15-d1cb-4d4c-af02-8888140971f9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '26439244d8a242598ddc188d87b97961', 'request_id': 'req-5fc46a15-d1cb-4d4c-af02-8888140971f9'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:33:44.918283 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-5fc46a15-d1cb-4d4c-af02-8888140971f9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '26439244d8a242598ddc188d87b97961', 'request_id': 'req-5fc46a15-d1cb-4d4c-af02-8888140971f9'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:33:44.973366 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-5fc46a15-d1cb-4d4c-af02-8888140971f9 admin admin] The token cache is being invalidate because project 26439244d8a242598ddc188d87b97961 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:33:44.973896 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-5fc46a15-d1cb-4d4c-af02-8888140971f9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:33:44.979781 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-5fc46a15-d1cb-4d4c-af02-8888140971f9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:33:44.998695 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 824/1649] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:33:44 2022] DELETE /identity/v3/projects/26439244d8a242598ddc188d87b97961 => generated 0 bytes in 505 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:33:45.023860 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fc46a15-d1cb-4d4c-af02-8888140971f9 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:45.030247 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-5fc46a15-d1cb-4d4c-af02-8888140971f9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:33:45.060672 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 826/1650] 10.222.0.90 () {58 vars in 1018 bytes} [Mon Aug 29 21:33:44 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 321 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:33:45.114514 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45c4424f-3c1a-468e-93a1-5f3ee79a04eb None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:45.116243 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-45c4424f-3c1a-468e-93a1-5f3ee79a04eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:33:45.180894 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45c4424f-3c1a-468e-93a1-5f3ee79a04eb None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:45.219948 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70ce9a14-6335-47ac-b896-f5eac86ead83 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b95fa07a7c0549dbaac49406e2639e21', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:45.220903 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70ce9a14-6335-47ac-b896-f5eac86ead83 service placement] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:45.221424 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70ce9a14-6335-47ac-b896-f5eac86ead83 service placement] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:45.221937 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70ce9a14-6335-47ac-b896-f5eac86ead83 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:45.259456 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70ce9a14-6335-47ac-b896-f5eac86ead83 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:45.267585 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70ce9a14-6335-47ac-b896-f5eac86ead83 service placement] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:45.287823 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 827/1651] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:33:45 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 174 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 21:33:45.326787 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fc46a15-d1cb-4d4c-af02-8888140971f9 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:45.341042 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-882555c9-4485-4ce7-a2ba-c009d6326fb2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:45.342293 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-882555c9-4485-4ce7-a2ba-c009d6326fb2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:45.342838 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-882555c9-4485-4ce7-a2ba-c009d6326fb2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:45.343411 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-882555c9-4485-4ce7-a2ba-c009d6326fb2 admin admin] PATH_INFO: `/v3/users/8bdb7810412d409c9e8aa202ceb6463c` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:45.345565 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-882555c9-4485-4ce7-a2ba-c009d6326fb2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8bdb7810412d409c9e8aa202ceb6463c)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:45.349180 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-882555c9-4485-4ce7-a2ba-c009d6326fb2 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:45.613762 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-882555c9-4485-4ce7-a2ba-c009d6326fb2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8bdb7810412d409c9e8aa202ceb6463c'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:33:45.640997 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-882555c9-4485-4ce7-a2ba-c009d6326fb2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8bdb7810412d409c9e8aa202ceb6463c'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:33:45.641815 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-882555c9-4485-4ce7-a2ba-c009d6326fb2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8bdb7810412d409c9e8aa202ceb6463c'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:33:45.642466 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-882555c9-4485-4ce7-a2ba-c009d6326fb2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8bdb7810412d409c9e8aa202ceb6463c'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:33:45.652870 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-882555c9-4485-4ce7-a2ba-c009d6326fb2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8bdb7810412d409c9e8aa202ceb6463c'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:33:45.688787 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 825/1652] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:33:45 2022] DELETE /identity/v3/users/8bdb7810412d409c9e8aa202ceb6463c => generated 0 bytes in 674 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:33:46.466422 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12dddced-87a1-4a56-a3a6-b087081363d1 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:46.469000 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12dddced-87a1-4a56-a3a6-b087081363d1 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:46.469752 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12dddced-87a1-4a56-a3a6-b087081363d1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:46.607555 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-12dddced-87a1-4a56-a3a6-b087081363d1 None None] MFA Rules not processed for user `39f161e61a514d94b6008bd006cda5ce`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:33:46.610155 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-12dddced-87a1-4a56-a3a6-b087081363d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:33:46.724318 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-882555c9-4485-4ce7-a2ba-c009d6326fb2 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:46.728418 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-882555c9-4485-4ce7-a2ba-c009d6326fb2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:33:46.764642 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=96,keystone:INSERT=1,keystone:DELETE=3 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:33:46.906904 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 828/1653] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:33:46 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 446 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:33:46.911813 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-addb880b-3a08-4384-836a-cd092ff2a13c None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:46.912492 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-addb880b-3a08-4384-836a-cd092ff2a13c None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:46.914595 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-addb880b-3a08-4384-836a-cd092ff2a13c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:46.938965 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:DELETE=15,keystone:SELECT=83,keystone:INSERT=2 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:33:46.983819 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-addb880b-3a08-4384-836a-cd092ff2a13c None None] MFA Rules not processed for user `39f161e61a514d94b6008bd006cda5ce`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:33:46.985286 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-addb880b-3a08-4384-836a-cd092ff2a13c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:33:46.994815 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-882555c9-4485-4ce7-a2ba-c009d6326fb2 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:47.003998 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b5fc1dc-6b3f-4193-a907-6399a6722208 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:47.005969 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b5fc1dc-6b3f-4193-a907-6399a6722208 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:47.007129 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b5fc1dc-6b3f-4193-a907-6399a6722208 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:47.008639 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b5fc1dc-6b3f-4193-a907-6399a6722208 admin admin] PATH_INFO: `/v3/users/c2f53bb29a90436782679aed93fa2c49` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:47.011928 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b5fc1dc-6b3f-4193-a907-6399a6722208 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c2f53bb29a90436782679aed93fa2c49)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:47.019595 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b5fc1dc-6b3f-4193-a907-6399a6722208 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:47.020369 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 829/1654] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:33:46 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:33:47.040041 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-addb880b-3a08-4384-836a-cd092ff2a13c None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:47.042063 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-addb880b-3a08-4384-836a-cd092ff2a13c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:33:47.191100 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-7b5fc1dc-6b3f-4193-a907-6399a6722208 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c2f53bb29a90436782679aed93fa2c49'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:33:47.224539 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-7b5fc1dc-6b3f-4193-a907-6399a6722208 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c2f53bb29a90436782679aed93fa2c49'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:33:47.225318 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-7b5fc1dc-6b3f-4193-a907-6399a6722208 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c2f53bb29a90436782679aed93fa2c49'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:33:47.226019 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-7b5fc1dc-6b3f-4193-a907-6399a6722208 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c2f53bb29a90436782679aed93fa2c49'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:33:47.250649 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-7b5fc1dc-6b3f-4193-a907-6399a6722208 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c2f53bb29a90436782679aed93fa2c49'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:33:47.284761 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 826/1655] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:33:46 2022] DELETE /identity/v3/users/c2f53bb29a90436782679aed93fa2c49 => generated 0 bytes in 561 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 21:33:47.291788 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed4e3c29-0231-4f1e-afe0-80cf1b435eb5 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:47.292717 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed4e3c29-0231-4f1e-afe0-80cf1b435eb5 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:47.294335 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed4e3c29-0231-4f1e-afe0-80cf1b435eb5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:47.310865 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-addb880b-3a08-4384-836a-cd092ff2a13c None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:47.399609 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-ed4e3c29-0231-4f1e-afe0-80cf1b435eb5 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:33:47.403376 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-ed4e3c29-0231-4f1e-afe0-80cf1b435eb5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:33:47.467855 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8d82dee-b2c2-45db-8596-83a865fb6529 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a35b5b7dea7b421da056f063d2436ffb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:47.468737 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8d82dee-b2c2-45db-8596-83a865fb6529 service nova] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:47.469104 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8d82dee-b2c2-45db-8596-83a865fb6529 service nova] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:47.469492 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8d82dee-b2c2-45db-8596-83a865fb6529 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:47.477456 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-b8d82dee-b2c2-45db-8596-83a865fb6529 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:33:47.588169 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 827/1656] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:33:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 299 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:33:47.677204 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8d82dee-b2c2-45db-8596-83a865fb6529 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:47.680271 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8d82dee-b2c2-45db-8596-83a865fb6529 service nova] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:47.772224 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 830/1657] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:33:47 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 733 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 21:33:47.884294 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed4e3c29-0231-4f1e-afe0-80cf1b435eb5 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:47.896695 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-ed4e3c29-0231-4f1e-afe0-80cf1b435eb5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:33:47.923347 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed4e3c29-0231-4f1e-afe0-80cf1b435eb5 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:47.969595 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56f46186-0a08-42fb-82a2-0125321e61aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:47.974918 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f46186-0a08-42fb-82a2-0125321e61aa admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:47.977466 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f46186-0a08-42fb-82a2-0125321e61aa admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:47.979235 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f46186-0a08-42fb-82a2-0125321e61aa admin admin] PATH_INFO: `/v3/users/691ad2f5ff824b9490cd18024edfe508` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:47.987974 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56f46186-0a08-42fb-82a2-0125321e61aa admin admin] RBAC: Authorizing `identity:delete_user(user_id=691ad2f5ff824b9490cd18024edfe508)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:48.002470 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56f46186-0a08-42fb-82a2-0125321e61aa admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:48.138542 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8d82dee-b2c2-45db-8596-83a865fb6529 service nova] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:48.140139 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-b8d82dee-b2c2-45db-8596-83a865fb6529 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:33:48.161824 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:33:48.170077 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-56f46186-0a08-42fb-82a2-0125321e61aa admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '691ad2f5ff824b9490cd18024edfe508'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:33:48.182361 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=81,keystone:DELETE=18,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:33:48.192973 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-56f46186-0a08-42fb-82a2-0125321e61aa admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '691ad2f5ff824b9490cd18024edfe508'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:33:48.195682 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-56f46186-0a08-42fb-82a2-0125321e61aa admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '691ad2f5ff824b9490cd18024edfe508'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:33:48.197914 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-56f46186-0a08-42fb-82a2-0125321e61aa admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '691ad2f5ff824b9490cd18024edfe508'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:33:48.222315 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-56f46186-0a08-42fb-82a2-0125321e61aa admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '691ad2f5ff824b9490cd18024edfe508'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:33:48.262629 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 828/1658] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:33:47 2022] DELETE /identity/v3/users/691ad2f5ff824b9490cd18024edfe508 => generated 0 bytes in 380 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:33:48.314547 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8d82dee-b2c2-45db-8596-83a865fb6529 service nova] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:48.434045 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-783f652e-cf62-4f83-b707-03fde8f84ca1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:48.434833 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-783f652e-cf62-4f83-b707-03fde8f84ca1 service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:48.435123 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-783f652e-cf62-4f83-b707-03fde8f84ca1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:48.435399 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-783f652e-cf62-4f83-b707-03fde8f84ca1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:48.444100 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-783f652e-cf62-4f83-b707-03fde8f84ca1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:33:48.615422 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-783f652e-cf62-4f83-b707-03fde8f84ca1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:48.617335 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-783f652e-cf62-4f83-b707-03fde8f84ca1 service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:48.630283 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 831/1659] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:33:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 492 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:33:49.003950 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20b8f6db-520f-4781-b8df-44f2b2373217 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:49.004562 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20b8f6db-520f-4781-b8df-44f2b2373217 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:49.004818 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20b8f6db-520f-4781-b8df-44f2b2373217 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:49.006914 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 829/1660] 10.222.0.90 () {54 vars in 912 bytes} [Mon Aug 29 21:33:49 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:33:49.019498 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d096ff30-4777-47ee-b815-a1948ca851f1 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:49.019795 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d096ff30-4777-47ee-b815-a1948ca851f1 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:49.020081 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d096ff30-4777-47ee-b815-a1948ca851f1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:49.106528 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-d096ff30-4777-47ee-b815-a1948ca851f1 None None] MFA Rules not processed for user `0e69b7f0c3574abc9de5fb580d22152a`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:33:49.107996 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-d096ff30-4777-47ee-b815-a1948ca851f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:33:49.123867 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 832/1661] 10.222.0.90 () {58 vars in 1009 bytes} [Mon Aug 29 21:33:49 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 21:33:49.145192 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20b8f6db-520f-4781-b8df-44f2b2373217 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:49.147715 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-20b8f6db-520f-4781-b8df-44f2b2373217 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:33:49.176945 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d096ff30-4777-47ee-b815-a1948ca851f1 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:49.178231 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-d096ff30-4777-47ee-b815-a1948ca851f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:33:49.193949 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20b8f6db-520f-4781-b8df-44f2b2373217 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:49.206889 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d096ff30-4777-47ee-b815-a1948ca851f1 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:49.208134 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52b9c88e-d09b-4de2-8dde-f81926e27ea1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:49.208785 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52b9c88e-d09b-4de2-8dde-f81926e27ea1 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:49.209058 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52b9c88e-d09b-4de2-8dde-f81926e27ea1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:49.209341 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52b9c88e-d09b-4de2-8dde-f81926e27ea1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:49.226825 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52b9c88e-d09b-4de2-8dde-f81926e27ea1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:49.230655 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52b9c88e-d09b-4de2-8dde-f81926e27ea1 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:49.231641 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5e2259e-fe17-4218-ac07-1c5558356a9f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:49.232325 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5e2259e-fe17-4218-ac07-1c5558356a9f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:49.232612 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5e2259e-fe17-4218-ac07-1c5558356a9f admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:49.232894 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5e2259e-fe17-4218-ac07-1c5558356a9f admin admin] PATH_INFO: `/v3/projects/1973a2c77684405281681124fcf2a09c` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:49.242069 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5e2259e-fe17-4218-ac07-1c5558356a9f admin admin] RBAC: Authorizing `identity:delete_project(project_id=1973a2c77684405281681124fcf2a09c)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:49.244055 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5e2259e-fe17-4218-ac07-1c5558356a9f admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:49.269626 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 830/1662] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:33:49 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 125 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 21:33:49.390548 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-f5e2259e-fe17-4218-ac07-1c5558356a9f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1973a2c77684405281681124fcf2a09c', 'request_id': 'req-f5e2259e-fe17-4218-ac07-1c5558356a9f'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:33:49.399676 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=96,keystone:DELETE=4 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:33:49.417190 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-f5e2259e-fe17-4218-ac07-1c5558356a9f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1973a2c77684405281681124fcf2a09c', 'request_id': 'req-f5e2259e-fe17-4218-ac07-1c5558356a9f'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:33:49.433592 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-f5e2259e-fe17-4218-ac07-1c5558356a9f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1973a2c77684405281681124fcf2a09c', 'request_id': 'req-f5e2259e-fe17-4218-ac07-1c5558356a9f'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:33:49.496639 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-f5e2259e-fe17-4218-ac07-1c5558356a9f admin admin] The token cache is being invalidate because project 1973a2c77684405281681124fcf2a09c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81434) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:33:49.498485 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-f5e2259e-fe17-4218-ac07-1c5558356a9f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:33:49.501731 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-f5e2259e-fe17-4218-ac07-1c5558356a9f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:33:49.511847 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 833/1663] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:33:49 2022] DELETE /identity/v3/projects/1973a2c77684405281681124fcf2a09c => generated 0 bytes in 336 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:33:49.518643 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52b9c88e-d09b-4de2-8dde-f81926e27ea1 service neutron] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:49.551036 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52b9c88e-d09b-4de2-8dde-f81926e27ea1 service neutron] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:49.575929 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5e2259e-fe17-4218-ac07-1c5558356a9f admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:49.579607 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-f5e2259e-fe17-4218-ac07-1c5558356a9f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:33:49.759801 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9599685-1a10-47d7-9abc-8c60174ed2ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:49.762754 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9599685-1a10-47d7-9abc-8c60174ed2ba admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:49.763055 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9599685-1a10-47d7-9abc-8c60174ed2ba admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:49.763361 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9599685-1a10-47d7-9abc-8c60174ed2ba admin admin] PATH_INFO: `/v3/users/addde520faa7413b8f351e49d42dabb9` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:49.764690 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9599685-1a10-47d7-9abc-8c60174ed2ba admin admin] RBAC: Authorizing `identity:delete_user(user_id=addde520faa7413b8f351e49d42dabb9)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:49.770890 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9599685-1a10-47d7-9abc-8c60174ed2ba admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:49.849246 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5e2259e-fe17-4218-ac07-1c5558356a9f admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:49.871673 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f3a4110-364f-482e-9f44-19ed71d0b5bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:49.874296 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f3a4110-364f-482e-9f44-19ed71d0b5bb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:49.875982 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f3a4110-364f-482e-9f44-19ed71d0b5bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:49.876460 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f3a4110-364f-482e-9f44-19ed71d0b5bb admin admin] PATH_INFO: `/v3/projects/d686968826f340cfac2cf7bdb1bb8d58` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:49.880106 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f3a4110-364f-482e-9f44-19ed71d0b5bb admin admin] RBAC: Authorizing `identity:delete_project(project_id=d686968826f340cfac2cf7bdb1bb8d58)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:49.884202 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f3a4110-364f-482e-9f44-19ed71d0b5bb admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:49.978776 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-e9599685-1a10-47d7-9abc-8c60174ed2ba admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'addde520faa7413b8f351e49d42dabb9'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:33:50.027117 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-e9599685-1a10-47d7-9abc-8c60174ed2ba admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'addde520faa7413b8f351e49d42dabb9'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:33:50.029292 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-e9599685-1a10-47d7-9abc-8c60174ed2ba admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'addde520faa7413b8f351e49d42dabb9'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:33:50.030859 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-e9599685-1a10-47d7-9abc-8c60174ed2ba admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'addde520faa7413b8f351e49d42dabb9'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:33:50.031732 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-9f3a4110-364f-482e-9f44-19ed71d0b5bb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd686968826f340cfac2cf7bdb1bb8d58', 'request_id': 'req-9f3a4110-364f-482e-9f44-19ed71d0b5bb'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:33:50.062040 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-9f3a4110-364f-482e-9f44-19ed71d0b5bb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd686968826f340cfac2cf7bdb1bb8d58', 'request_id': 'req-9f3a4110-364f-482e-9f44-19ed71d0b5bb'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:33:50.063714 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-e9599685-1a10-47d7-9abc-8c60174ed2ba admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'addde520faa7413b8f351e49d42dabb9'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:33:50.080174 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-9f3a4110-364f-482e-9f44-19ed71d0b5bb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd686968826f340cfac2cf7bdb1bb8d58', 'request_id': 'req-9f3a4110-364f-482e-9f44-19ed71d0b5bb'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:33:50.105558 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 831/1664] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:33:49 2022] DELETE /identity/v3/users/addde520faa7413b8f351e49d42dabb9 => generated 0 bytes in 589 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:33:50.148742 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-9f3a4110-364f-482e-9f44-19ed71d0b5bb admin admin] The token cache is being invalidate because project d686968826f340cfac2cf7bdb1bb8d58 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81434) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:33:50.149676 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-9f3a4110-364f-482e-9f44-19ed71d0b5bb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:33:50.154065 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-9f3a4110-364f-482e-9f44-19ed71d0b5bb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:33:50.170094 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 834/1665] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:33:49 2022] DELETE /identity/v3/projects/d686968826f340cfac2cf7bdb1bb8d58 => generated 0 bytes in 588 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:33:50.171021 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e41b6725-f4a5-4a2d-9e0b-6b54720ac20a None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:50.171304 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e41b6725-f4a5-4a2d-9e0b-6b54720ac20a None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:50.171764 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e41b6725-f4a5-4a2d-9e0b-6b54720ac20a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:50.229471 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-e41b6725-f4a5-4a2d-9e0b-6b54720ac20a None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:33:50.239474 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-e41b6725-f4a5-4a2d-9e0b-6b54720ac20a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:33:50.363012 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:33:50.429952 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 832/1666] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:33:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 260 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 21:33:50.444508 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-693c1996-afaa-463a-9180-8c94784f9d67 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:50.456389 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-693c1996-afaa-463a-9180-8c94784f9d67 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:50.457196 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-693c1996-afaa-463a-9180-8c94784f9d67 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:50.541530 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-693c1996-afaa-463a-9180-8c94784f9d67 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:33:50.543406 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-693c1996-afaa-463a-9180-8c94784f9d67 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:33:50.619554 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 835/1667] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:33:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:33:50.621270 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e41b6725-f4a5-4a2d-9e0b-6b54720ac20a None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:50.636208 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-693c1996-afaa-463a-9180-8c94784f9d67 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:50.638832 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-693c1996-afaa-463a-9180-8c94784f9d67 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:33:50.643374 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e41b6725-f4a5-4a2d-9e0b-6b54720ac20a None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:50.664714 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a1d3740-ef13-46dc-94c1-080b1862ba39 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:50.666467 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a1d3740-ef13-46dc-94c1-080b1862ba39 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:50.667006 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a1d3740-ef13-46dc-94c1-080b1862ba39 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:50.667815 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a1d3740-ef13-46dc-94c1-080b1862ba39 admin admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:50.669282 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a1d3740-ef13-46dc-94c1-080b1862ba39 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:33:50.669852 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a1d3740-ef13-46dc-94c1-080b1862ba39 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:50.675440 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a1d3740-ef13-46dc-94c1-080b1862ba39 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:50.705955 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 833/1668] 10.222.0.90 () {58 vars in 1153 bytes} [Mon Aug 29 21:33:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:33:50.713740 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a1d3740-ef13-46dc-94c1-080b1862ba39 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:50.735905 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a1d3740-ef13-46dc-94c1-080b1862ba39 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:50.751075 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9454cc4a-fe50-4924-9d28-94eccd1f8399 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:50.752054 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9454cc4a-fe50-4924-9d28-94eccd1f8399 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:50.752523 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9454cc4a-fe50-4924-9d28-94eccd1f8399 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:50.752995 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9454cc4a-fe50-4924-9d28-94eccd1f8399 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:50.754403 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9454cc4a-fe50-4924-9d28-94eccd1f8399 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:50.758231 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9454cc4a-fe50-4924-9d28-94eccd1f8399 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:50.838895 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 834/1669] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:33:50 2022] POST /identity/v3/projects => generated 409 bytes in 126 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 21:33:50.844033 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9454cc4a-fe50-4924-9d28-94eccd1f8399 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:50.883268 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:DELETE=10,keystone:SELECT=88,keystone:INSERT=2 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:33:50.934995 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-693c1996-afaa-463a-9180-8c94784f9d67 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:51.011360 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9454cc4a-fe50-4924-9d28-94eccd1f8399 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:51.031747 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6443481-38b6-4f73-bd40-6c2c246d1ace admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:51.032636 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6443481-38b6-4f73-bd40-6c2c246d1ace admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:51.032636 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6443481-38b6-4f73-bd40-6c2c246d1ace admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:51.032894 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6443481-38b6-4f73-bd40-6c2c246d1ace admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:51.033735 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6443481-38b6-4f73-bd40-6c2c246d1ace admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:51.035870 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6443481-38b6-4f73-bd40-6c2c246d1ace admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:51.092106 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 835/1670] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:33:50 2022] POST /identity/v3/users => generated 305 bytes in 249 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 21:33:51.093634 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a58a760d-3969-4dc7-9994-67e1f6bce6a0 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:51.093971 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a58a760d-3969-4dc7-9994-67e1f6bce6a0 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:51.094249 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a58a760d-3969-4dc7-9994-67e1f6bce6a0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:51.149380 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-a58a760d-3969-4dc7-9994-67e1f6bce6a0 None None] MFA Rules not processed for user `39f161e61a514d94b6008bd006cda5ce`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:33:51.150870 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-a58a760d-3969-4dc7-9994-67e1f6bce6a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:33:51.190180 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e95e53a3-fb31-447e-a6af-0099843fc36e service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b95fa07a7c0549dbaac49406e2639e21', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:51.190819 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e95e53a3-fb31-447e-a6af-0099843fc36e service placement] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:51.191102 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e95e53a3-fb31-447e-a6af-0099843fc36e service placement] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:51.204184 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e95e53a3-fb31-447e-a6af-0099843fc36e service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:51.213422 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-e95e53a3-fb31-447e-a6af-0099843fc36e service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:33:51.244278 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e95e53a3-fb31-447e-a6af-0099843fc36e service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:51.252178 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e95e53a3-fb31-447e-a6af-0099843fc36e service placement] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:51.297210 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 836/1671] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:33:50 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 662 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:33:51.298606 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e95e53a3-fb31-447e-a6af-0099843fc36e service placement] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:51.305489 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-e95e53a3-fb31-447e-a6af-0099843fc36e service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:33:51.323485 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 836/1672] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:33:51 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 231 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:33:51.328581 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a58a760d-3969-4dc7-9994-67e1f6bce6a0 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:51.361657 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e95e53a3-fb31-447e-a6af-0099843fc36e service placement] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:51.364948 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a58a760d-3969-4dc7-9994-67e1f6bce6a0 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:51.368338 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffca8097-8050-4cd6-8933-c1e8bf1a4e30 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:51.369171 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffca8097-8050-4cd6-8933-c1e8bf1a4e30 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:51.369594 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffca8097-8050-4cd6-8933-c1e8bf1a4e30 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:51.370051 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffca8097-8050-4cd6-8933-c1e8bf1a4e30 admin admin] PATH_INFO: `/v3/projects/bce8b2975fab487c8bb797c80a8c7a4f` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:51.371344 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffca8097-8050-4cd6-8933-c1e8bf1a4e30 admin admin] RBAC: Authorizing `identity:delete_project(project_id=bce8b2975fab487c8bb797c80a8c7a4f)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:51.373805 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffca8097-8050-4cd6-8933-c1e8bf1a4e30 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:51.377348 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-511d48a4-e501-4342-8f83-cac6c2ed2409 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:51.378032 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-511d48a4-e501-4342-8f83-cac6c2ed2409 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:51.378320 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-511d48a4-e501-4342-8f83-cac6c2ed2409 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:51.378602 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-511d48a4-e501-4342-8f83-cac6c2ed2409 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:51.380391 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-511d48a4-e501-4342-8f83-cac6c2ed2409 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:51.382660 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-511d48a4-e501-4342-8f83-cac6c2ed2409 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:51.406405 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 837/1673] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:33:51 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:33:51.412813 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-511d48a4-e501-4342-8f83-cac6c2ed2409 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:51.422242 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-511d48a4-e501-4342-8f83-cac6c2ed2409 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:33:51.452464 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-511d48a4-e501-4342-8f83-cac6c2ed2409 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:51.487094 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-361e1edb-eb2d-4726-82c7-29efa0a2c69e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:51.488184 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-361e1edb-eb2d-4726-82c7-29efa0a2c69e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:51.488865 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-361e1edb-eb2d-4726-82c7-29efa0a2c69e admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:51.489156 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-361e1edb-eb2d-4726-82c7-29efa0a2c69e admin admin] PATH_INFO: `/v3/projects/3057e78899314a99bf88c55ab9b0fa25` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:51.500221 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-361e1edb-eb2d-4726-82c7-29efa0a2c69e admin admin] RBAC: Authorizing `identity:delete_project(project_id=3057e78899314a99bf88c55ab9b0fa25)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:51.519530 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-361e1edb-eb2d-4726-82c7-29efa0a2c69e admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:51.586884 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-ffca8097-8050-4cd6-8933-c1e8bf1a4e30 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bce8b2975fab487c8bb797c80a8c7a4f', 'request_id': 'req-ffca8097-8050-4cd6-8933-c1e8bf1a4e30'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:33:51.601513 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-ffca8097-8050-4cd6-8933-c1e8bf1a4e30 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bce8b2975fab487c8bb797c80a8c7a4f', 'request_id': 'req-ffca8097-8050-4cd6-8933-c1e8bf1a4e30'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:33:51.608323 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=95,keystone:INSERT=4,keystone:DELETE=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:33:51.628175 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-ffca8097-8050-4cd6-8933-c1e8bf1a4e30 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bce8b2975fab487c8bb797c80a8c7a4f', 'request_id': 'req-ffca8097-8050-4cd6-8933-c1e8bf1a4e30'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:33:51.661186 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-ffca8097-8050-4cd6-8933-c1e8bf1a4e30 admin admin] The token cache is being invalidate because project bce8b2975fab487c8bb797c80a8c7a4f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81434) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:33:51.661537 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-ffca8097-8050-4cd6-8933-c1e8bf1a4e30 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:33:51.663552 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-ffca8097-8050-4cd6-8933-c1e8bf1a4e30 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:33:51.664067 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-361e1edb-eb2d-4726-82c7-29efa0a2c69e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3057e78899314a99bf88c55ab9b0fa25', 'request_id': 'req-361e1edb-eb2d-4726-82c7-29efa0a2c69e'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:33:51.666961 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 837/1674] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:33:51 2022] DELETE /identity/v3/projects/bce8b2975fab487c8bb797c80a8c7a4f => generated 0 bytes in 369 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:33:51.673258 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffca8097-8050-4cd6-8933-c1e8bf1a4e30 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:51.679551 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-ffca8097-8050-4cd6-8933-c1e8bf1a4e30 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:33:51.688753 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-361e1edb-eb2d-4726-82c7-29efa0a2c69e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3057e78899314a99bf88c55ab9b0fa25', 'request_id': 'req-361e1edb-eb2d-4726-82c7-29efa0a2c69e'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:33:51.702535 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-361e1edb-eb2d-4726-82c7-29efa0a2c69e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3057e78899314a99bf88c55ab9b0fa25', 'request_id': 'req-361e1edb-eb2d-4726-82c7-29efa0a2c69e'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:33:51.745106 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-361e1edb-eb2d-4726-82c7-29efa0a2c69e admin admin] The token cache is being invalidate because project 3057e78899314a99bf88c55ab9b0fa25 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:33:51.746473 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-361e1edb-eb2d-4726-82c7-29efa0a2c69e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:33:51.751624 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-361e1edb-eb2d-4726-82c7-29efa0a2c69e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:33:51.761411 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 838/1675] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:33:51 2022] DELETE /identity/v3/projects/3057e78899314a99bf88c55ab9b0fa25 => generated 0 bytes in 350 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:33:51.763103 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb832627-10c9-496f-8264-cc4b78e458a9 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:51.765522 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb832627-10c9-496f-8264-cc4b78e458a9 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:51.765940 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb832627-10c9-496f-8264-cc4b78e458a9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:51.826183 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-fb832627-10c9-496f-8264-cc4b78e458a9 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:33:51.828616 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-fb832627-10c9-496f-8264-cc4b78e458a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:33:51.911178 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffca8097-8050-4cd6-8933-c1e8bf1a4e30 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:51.984218 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:33:52.057408 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 839/1676] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:33:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 295 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:33:52.058988 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a35bf98f-2be3-4684-b4c5-25d016e538e9 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:52.061728 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a35bf98f-2be3-4684-b4c5-25d016e538e9 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:52.062203 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a35bf98f-2be3-4684-b4c5-25d016e538e9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:52.128225 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-a35bf98f-2be3-4684-b4c5-25d016e538e9 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:33:52.130988 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-a35bf98f-2be3-4684-b4c5-25d016e538e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:33:52.142379 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8817b264-a654-440c-a799-8237c0202ecd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:52.144041 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8817b264-a654-440c-a799-8237c0202ecd admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:52.145688 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8817b264-a654-440c-a799-8237c0202ecd admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:52.146002 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8817b264-a654-440c-a799-8237c0202ecd admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:52.150840 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8817b264-a654-440c-a799-8237c0202ecd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:52.156542 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8817b264-a654-440c-a799-8237c0202ecd admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:52.164242 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 840/1677] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:33:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:33:52.167308 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ed00722-899c-4e6f-bbaf-d903d51db649 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:52.167647 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ed00722-899c-4e6f-bbaf-d903d51db649 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:52.167927 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ed00722-899c-4e6f-bbaf-d903d51db649 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:52.195505 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 838/1678] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:33:51 2022] GET /identity/v3/roles => generated 1420 bytes in 528 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:33:52.198398 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-054b4d5f-3721-454d-9b3b-91869c7616bd None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:52.198994 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-054b4d5f-3721-454d-9b3b-91869c7616bd None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:52.199501 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-054b4d5f-3721-454d-9b3b-91869c7616bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:52.231618 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-054b4d5f-3721-454d-9b3b-91869c7616bd None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:33:52.233219 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-054b4d5f-3721-454d-9b3b-91869c7616bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:33:52.270468 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 839/1679] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:33:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 73 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:33:52.273322 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edd5a50e-8d97-4874-bc8e-959f59c78b90 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:52.278183 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edd5a50e-8d97-4874-bc8e-959f59c78b90 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:52.278682 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edd5a50e-8d97-4874-bc8e-959f59c78b90 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:52.289082 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-8ed00722-899c-4e6f-bbaf-d903d51db649 None None] MFA Rules not processed for user `b33a740f1c8a4d9db8d199eff66b76b2`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:33:52.290820 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-8ed00722-899c-4e6f-bbaf-d903d51db649 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:33:52.369732 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-edd5a50e-8d97-4874-bc8e-959f59c78b90 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:33:52.370776 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-edd5a50e-8d97-4874-bc8e-959f59c78b90 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:33:52.402182 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 840/1680] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:33:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:33:52.407789 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edd5a50e-8d97-4874-bc8e-959f59c78b90 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:52.412656 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-edd5a50e-8d97-4874-bc8e-959f59c78b90 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:33:52.467368 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edd5a50e-8d97-4874-bc8e-959f59c78b90 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:52.502174 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dbe28f0-a0f2-4f2d-a69e-a07f98c13635 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:52.502887 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dbe28f0-a0f2-4f2d-a69e-a07f98c13635 admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:52.503181 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dbe28f0-a0f2-4f2d-a69e-a07f98c13635 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:52.503554 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dbe28f0-a0f2-4f2d-a69e-a07f98c13635 admin admin] PATH_INFO: `/v3/projects/e02a98407a4b4cc186409f1a8768abfe/users/abbce453b48243e28acb1a80df49f513/roles/02e242d5520740d4b77a5f4008c7eb57` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:52.524905 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 841/1681] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:33:52 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 359 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:33:52.528077 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ed00722-899c-4e6f-bbaf-d903d51db649 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:52.556503 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ed00722-899c-4e6f-bbaf-d903d51db649 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:52.568609 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bed968b-b3a8-4a22-a65a-0ea18c657404 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:52.571044 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bed968b-b3a8-4a22-a65a-0ea18c657404 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:52.571556 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bed968b-b3a8-4a22-a65a-0ea18c657404 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:52.571890 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bed968b-b3a8-4a22-a65a-0ea18c657404 admin admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:52.572603 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bed968b-b3a8-4a22-a65a-0ea18c657404 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:33:52.572914 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bed968b-b3a8-4a22-a65a-0ea18c657404 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:52.580395 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bed968b-b3a8-4a22-a65a-0ea18c657404 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:52.617528 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 842/1682] 10.222.0.90 () {58 vars in 1153 bytes} [Mon Aug 29 21:33:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 92 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:33:52.619029 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bed968b-b3a8-4a22-a65a-0ea18c657404 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:52.619234 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dbe28f0-a0f2-4f2d-a69e-a07f98c13635 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e02a98407a4b4cc186409f1a8768abfe, user_id=abbce453b48243e28acb1a80df49f513, role_id=02e242d5520740d4b77a5f4008c7eb57)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:52.621605 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-3bed968b-b3a8-4a22-a65a-0ea18c657404 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:33:52.627591 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dbe28f0-a0f2-4f2d-a69e-a07f98c13635 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:52.660217 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bed968b-b3a8-4a22-a65a-0ea18c657404 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:52.686620 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 841/1683] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:33:52 2022] PUT /identity/v3/projects/e02a98407a4b4cc186409f1a8768abfe/users/abbce453b48243e28acb1a80df49f513/roles/02e242d5520740d4b77a5f4008c7eb57 => generated 0 bytes in 284 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:33:52.693227 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dbe28f0-a0f2-4f2d-a69e-a07f98c13635 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:52.750886 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=93,keystone:DELETE=6,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:33:52.857249 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b86fc408-0d2c-4f9c-b239-80e8e1bfbbd6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:52.858310 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b86fc408-0d2c-4f9c-b239-80e8e1bfbbd6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:52.858777 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b86fc408-0d2c-4f9c-b239-80e8e1bfbbd6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:52.859206 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b86fc408-0d2c-4f9c-b239-80e8e1bfbbd6 admin admin] PATH_INFO: `/v3/users/2eabcdd8526f488291dbb237018c29ae` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:52.861987 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b86fc408-0d2c-4f9c-b239-80e8e1bfbbd6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2eabcdd8526f488291dbb237018c29ae)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:52.866087 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b86fc408-0d2c-4f9c-b239-80e8e1bfbbd6 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:52.958294 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dbe28f0-a0f2-4f2d-a69e-a07f98c13635 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:52.984158 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b5d87e5-a35e-453a-bad3-b9dc15d58001 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:52.985040 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b5d87e5-a35e-453a-bad3-b9dc15d58001 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:52.985458 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b5d87e5-a35e-453a-bad3-b9dc15d58001 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:52.986213 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b5d87e5-a35e-453a-bad3-b9dc15d58001 admin admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:52.987004 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b5d87e5-a35e-453a-bad3-b9dc15d58001 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:33:52.988257 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b5d87e5-a35e-453a-bad3-b9dc15d58001 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:52.991603 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b5d87e5-a35e-453a-bad3-b9dc15d58001 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:53.030084 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 842/1684] 10.222.0.90 () {58 vars in 1153 bytes} [Mon Aug 29 21:33:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 336 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:33:53.030084 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b5d87e5-a35e-453a-bad3-b9dc15d58001 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:53.037094 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-6b5d87e5-a35e-453a-bad3-b9dc15d58001 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:33:53.082047 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-b86fc408-0d2c-4f9c-b239-80e8e1bfbbd6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2eabcdd8526f488291dbb237018c29ae'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:33:53.124633 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-b86fc408-0d2c-4f9c-b239-80e8e1bfbbd6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2eabcdd8526f488291dbb237018c29ae'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:33:53.125672 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-b86fc408-0d2c-4f9c-b239-80e8e1bfbbd6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2eabcdd8526f488291dbb237018c29ae'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:33:53.127755 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-b86fc408-0d2c-4f9c-b239-80e8e1bfbbd6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2eabcdd8526f488291dbb237018c29ae'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:33:53.149935 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-b86fc408-0d2c-4f9c-b239-80e8e1bfbbd6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2eabcdd8526f488291dbb237018c29ae'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:33:53.184380 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 843/1685] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:33:52 2022] DELETE /identity/v3/users/2eabcdd8526f488291dbb237018c29ae => generated 0 bytes in 566 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:33:53.192361 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b86fc408-0d2c-4f9c-b239-80e8e1bfbbd6 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:53.198046 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-b86fc408-0d2c-4f9c-b239-80e8e1bfbbd6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:33:53.367122 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b5d87e5-a35e-453a-bad3-b9dc15d58001 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:53.387813 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b86fc408-0d2c-4f9c-b239-80e8e1bfbbd6 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:53.414371 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e240375f-c69f-4d87-979e-dcfd917ce1f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:53.415260 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e240375f-c69f-4d87-979e-dcfd917ce1f1 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:53.415573 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e240375f-c69f-4d87-979e-dcfd917ce1f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:53.415857 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e240375f-c69f-4d87-979e-dcfd917ce1f1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:53.416800 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e240375f-c69f-4d87-979e-dcfd917ce1f1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:53.425259 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:33:53.436243 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e240375f-c69f-4d87-979e-dcfd917ce1f1 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:53.570255 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 844/1686] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:33:53 2022] POST /identity/v3/projects => generated 355 bytes in 381 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:33:53.583814 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14cffe95-253f-445a-8e72-e44191a54522 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:53.584930 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14cffe95-253f-445a-8e72-e44191a54522 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:53.587026 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14cffe95-253f-445a-8e72-e44191a54522 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:53.663134 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-021753a6-b445-4069-8d05-dcbc0c9763dd service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a35b5b7dea7b421da056f063d2436ffb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:53.664755 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-021753a6-b445-4069-8d05-dcbc0c9763dd service nova] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:53.665193 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-021753a6-b445-4069-8d05-dcbc0c9763dd service nova] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:53.665591 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-021753a6-b445-4069-8d05-dcbc0c9763dd service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:53.672806 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-14cffe95-253f-445a-8e72-e44191a54522 None None] MFA Rules not processed for user `abbce453b48243e28acb1a80df49f513`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:33:53.674008 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-021753a6-b445-4069-8d05-dcbc0c9763dd service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:33:53.677218 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-14cffe95-253f-445a-8e72-e44191a54522 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:33:53.731492 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:33:53.906693 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-021753a6-b445-4069-8d05-dcbc0c9763dd service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:53.908665 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-021753a6-b445-4069-8d05-dcbc0c9763dd service nova] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:53.947428 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 843/1687] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:33:53 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 918 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:33:53.950198 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-021753a6-b445-4069-8d05-dcbc0c9763dd service nova] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:53.956621 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-021753a6-b445-4069-8d05-dcbc0c9763dd service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:33:53.985729 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 845/1688] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:33:53 2022] POST /identity/v3/auth/tokens => generated 3286 bytes in 409 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:33:54.001606 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07c90e61-a485-4d98-8b21-b8446048efd7 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:54.006065 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07c90e61-a485-4d98-8b21-b8446048efd7 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:54.006617 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07c90e61-a485-4d98-8b21-b8446048efd7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:54.035806 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-021753a6-b445-4069-8d05-dcbc0c9763dd service nova] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:54.115073 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-07c90e61-a485-4d98-8b21-b8446048efd7 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:33:54.124818 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-07c90e61-a485-4d98-8b21-b8446048efd7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:33:54.523875 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b73a3373-f708-4bca-bd9f-30aa5bce2e3b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:54.524799 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b73a3373-f708-4bca-bd9f-30aa5bce2e3b admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:54.525286 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b73a3373-f708-4bca-bd9f-30aa5bce2e3b admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:54.525823 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b73a3373-f708-4bca-bd9f-30aa5bce2e3b admin admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:54.541289 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b73a3373-f708-4bca-bd9f-30aa5bce2e3b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:54.545987 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b73a3373-f708-4bca-bd9f-30aa5bce2e3b admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:54.611426 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 846/1689] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:33:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 621 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:33:54.613490 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07c90e61-a485-4d98-8b21-b8446048efd7 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:54.647475 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 844/1690] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:33:53 2022] POST /identity/v3/projects => generated 373 bytes in 699 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:33:54.651572 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b73a3373-f708-4bca-bd9f-30aa5bce2e3b admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:54.653751 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07c90e61-a485-4d98-8b21-b8446048efd7 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:54.660359 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-b73a3373-f708-4bca-bd9f-30aa5bce2e3b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:33:54.867341 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:33:54.907549 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6355713c-3564-462b-a42d-8c32950835c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:54.908349 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6355713c-3564-462b-a42d-8c32950835c5 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:54.909725 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6355713c-3564-462b-a42d-8c32950835c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:54.910176 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6355713c-3564-462b-a42d-8c32950835c5 admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:54.910884 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6355713c-3564-462b-a42d-8c32950835c5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:54.916924 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6355713c-3564-462b-a42d-8c32950835c5 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:54.961238 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 847/1691] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:33:54 2022] POST /identity/v3/users => generated 278 bytes in 349 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:33:54.968708 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6355713c-3564-462b-a42d-8c32950835c5 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:54.998850 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6355713c-3564-462b-a42d-8c32950835c5 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:55.010870 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e0ac81f-fe32-4a12-9d8a-3621161c27b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:55.013243 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e0ac81f-fe32-4a12-9d8a-3621161c27b7 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:55.013742 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e0ac81f-fe32-4a12-9d8a-3621161c27b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:55.015533 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e0ac81f-fe32-4a12-9d8a-3621161c27b7 admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:55.016093 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b73a3373-f708-4bca-bd9f-30aa5bce2e3b admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:55.018320 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e0ac81f-fe32-4a12-9d8a-3621161c27b7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:55.025389 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e0ac81f-fe32-4a12-9d8a-3621161c27b7 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:55.042328 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be6a4493-e045-475d-b7e3-fd22cc4cd247 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:55.043354 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be6a4493-e045-475d-b7e3-fd22cc4cd247 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:55.043832 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be6a4493-e045-475d-b7e3-fd22cc4cd247 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:55.044300 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be6a4493-e045-475d-b7e3-fd22cc4cd247 admin admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:55.046084 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be6a4493-e045-475d-b7e3-fd22cc4cd247 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:55.053132 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be6a4493-e045-475d-b7e3-fd22cc4cd247 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:55.092225 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 848/1692] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:33:54 2022] POST /identity/v3/users => generated 287 bytes in 130 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:33:55.095048 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e0ac81f-fe32-4a12-9d8a-3621161c27b7 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:55.097996 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-8e0ac81f-fe32-4a12-9d8a-3621161c27b7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:33:55.165971 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 845/1693] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:33:54 2022] POST /identity/v3/projects => generated 409 bytes in 518 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:33:55.167475 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be6a4493-e045-475d-b7e3-fd22cc4cd247 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:55.191171 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:33:55.327170 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e0ac81f-fe32-4a12-9d8a-3621161c27b7 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:55.452761 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be6a4493-e045-475d-b7e3-fd22cc4cd247 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:55.473193 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50aaa66d-a511-4a99-af81-6aa5b7a4b2d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:55.473855 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50aaa66d-a511-4a99-af81-6aa5b7a4b2d5 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:55.474142 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50aaa66d-a511-4a99-af81-6aa5b7a4b2d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:55.474417 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50aaa66d-a511-4a99-af81-6aa5b7a4b2d5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:55.475080 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50aaa66d-a511-4a99-af81-6aa5b7a4b2d5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:55.477093 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50aaa66d-a511-4a99-af81-6aa5b7a4b2d5 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:55.520035 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 846/1694] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:33:55 2022] GET /identity/v3/roles => generated 1420 bytes in 354 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:33:55.535313 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3942a20a-c8f7-46cc-940c-41497cb28851 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:55.536301 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3942a20a-c8f7-46cc-940c-41497cb28851 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:55.537805 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3942a20a-c8f7-46cc-940c-41497cb28851 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:55.538366 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3942a20a-c8f7-46cc-940c-41497cb28851 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:55.541069 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50aaa66d-a511-4a99-af81-6aa5b7a4b2d5 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:55.577671 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50aaa66d-a511-4a99-af81-6aa5b7a4b2d5 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:55.581652 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3942a20a-c8f7-46cc-940c-41497cb28851 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:55.583692 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3942a20a-c8f7-46cc-940c-41497cb28851 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:55.613533 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 849/1695] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:33:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 521 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:33:55.615251 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3942a20a-c8f7-46cc-940c-41497cb28851 service neutron] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:55.622441 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ad6033e-6f3d-4771-bb91-97ed9eca7eeb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:55.624860 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ad6033e-6f3d-4771-bb91-97ed9eca7eeb admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:55.625263 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ad6033e-6f3d-4771-bb91-97ed9eca7eeb admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:55.625686 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ad6033e-6f3d-4771-bb91-97ed9eca7eeb admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:55.630924 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ad6033e-6f3d-4771-bb91-97ed9eca7eeb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:55.638975 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ad6033e-6f3d-4771-bb91-97ed9eca7eeb admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:55.644124 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3942a20a-c8f7-46cc-940c-41497cb28851 service neutron] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:55.664596 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bf99166-b660-4073-a320-80781da0bf09 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:55.665693 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bf99166-b660-4073-a320-80781da0bf09 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:55.666223 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bf99166-b660-4073-a320-80781da0bf09 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:55.666706 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bf99166-b660-4073-a320-80781da0bf09 admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:55.667602 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bf99166-b660-4073-a320-80781da0bf09 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:55.673825 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bf99166-b660-4073-a320-80781da0bf09 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:55.691967 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 847/1696] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:33:55 2022] GET /identity/v3/roles => generated 1420 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:33:55.695457 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ad6033e-6f3d-4771-bb91-97ed9eca7eeb admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:55.728934 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ad6033e-6f3d-4771-bb91-97ed9eca7eeb admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:55.739194 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 850/1697] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:33:55 2022] POST /identity/v3/users => generated 311 bytes in 124 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:33:55.743179 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bf99166-b660-4073-a320-80781da0bf09 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:55.749265 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a625aac-2b67-407f-afb5-420d86379893 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:55.750120 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a625aac-2b67-407f-afb5-420d86379893 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:55.753721 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a625aac-2b67-407f-afb5-420d86379893 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:55.754175 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a625aac-2b67-407f-afb5-420d86379893 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:55.755100 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a625aac-2b67-407f-afb5-420d86379893 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:55.761504 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a625aac-2b67-407f-afb5-420d86379893 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:55.788454 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bf99166-b660-4073-a320-80781da0bf09 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:55.806927 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 848/1698] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:33:55 2022] GET /identity/v3/roles => generated 1420 bytes in 114 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:33:55.809861 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a625aac-2b67-407f-afb5-420d86379893 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:55.811455 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd227d0f-1ec0-4e33-adcb-b5efea5156e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:55.812661 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd227d0f-1ec0-4e33-adcb-b5efea5156e1 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:55.818165 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd227d0f-1ec0-4e33-adcb-b5efea5156e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:55.819136 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd227d0f-1ec0-4e33-adcb-b5efea5156e1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:55.820807 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd227d0f-1ec0-4e33-adcb-b5efea5156e1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:55.824107 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd227d0f-1ec0-4e33-adcb-b5efea5156e1 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:55.852856 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a625aac-2b67-407f-afb5-420d86379893 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:55.877154 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 851/1699] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:33:55 2022] GET /identity/v3/roles => generated 1420 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:33:55.894087 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3561d52c-2b5e-47bb-9208-ed1c00ee487a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:55.894087 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3561d52c-2b5e-47bb-9208-ed1c00ee487a admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:55.894087 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3561d52c-2b5e-47bb-9208-ed1c00ee487a admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:55.894087 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3561d52c-2b5e-47bb-9208-ed1c00ee487a admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:55.894627 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3561d52c-2b5e-47bb-9208-ed1c00ee487a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:55.896823 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd227d0f-1ec0-4e33-adcb-b5efea5156e1 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:55.907744 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3561d52c-2b5e-47bb-9208-ed1c00ee487a admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:55.949379 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd227d0f-1ec0-4e33-adcb-b5efea5156e1 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:55.980868 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 849/1700] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:33:55 2022] GET /identity/v3/roles => generated 1420 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:33:55.981157 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47326410-001d-4812-b78e-305db0dc3bd9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:55.981329 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3561d52c-2b5e-47bb-9208-ed1c00ee487a admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:55.981475 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47326410-001d-4812-b78e-305db0dc3bd9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:55.981475 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47326410-001d-4812-b78e-305db0dc3bd9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:55.981475 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47326410-001d-4812-b78e-305db0dc3bd9 admin admin] PATH_INFO: `/v3/projects/9ac4f442da4a45a5b54b5fe400320b9a/users/704501227d4c4de9821fb2f427aaffad/roles/02e242d5520740d4b77a5f4008c7eb57` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:56.001060 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3561d52c-2b5e-47bb-9208-ed1c00ee487a admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:56.013356 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b661687-7806-4e6f-89f4-77fe8eb48984 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:56.031299 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b661687-7806-4e6f-89f4-77fe8eb48984 admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:56.032067 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b661687-7806-4e6f-89f4-77fe8eb48984 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:56.033059 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b661687-7806-4e6f-89f4-77fe8eb48984 admin admin] PATH_INFO: `/v3/projects/4ce47d12b3aa4d618ae4880856c5093f/users/71b71ed8570941f9946baad3cc7e7fba/roles/02e242d5520740d4b77a5f4008c7eb57` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:56.063790 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47326410-001d-4812-b78e-305db0dc3bd9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9ac4f442da4a45a5b54b5fe400320b9a, user_id=704501227d4c4de9821fb2f427aaffad, role_id=02e242d5520740d4b77a5f4008c7eb57)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:56.069309 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47326410-001d-4812-b78e-305db0dc3bd9 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:56.095637 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b661687-7806-4e6f-89f4-77fe8eb48984 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4ce47d12b3aa4d618ae4880856c5093f, user_id=71b71ed8570941f9946baad3cc7e7fba, role_id=02e242d5520740d4b77a5f4008c7eb57)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:56.114011 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b661687-7806-4e6f-89f4-77fe8eb48984 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:56.160547 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 852/1701] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:33:55 2022] PUT /identity/v3/projects/9ac4f442da4a45a5b54b5fe400320b9a/users/704501227d4c4de9821fb2f427aaffad/roles/02e242d5520740d4b77a5f4008c7eb57 => generated 0 bytes in 265 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:33:56.178151 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47326410-001d-4812-b78e-305db0dc3bd9 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:56.196191 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 850/1702] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:33:55 2022] PUT /identity/v3/projects/4ce47d12b3aa4d618ae4880856c5093f/users/71b71ed8570941f9946baad3cc7e7fba/roles/02e242d5520740d4b77a5f4008c7eb57 => generated 0 bytes in 193 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:33:56.201562 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9ea4c36-c2a8-4cc8-9b95-b35eada0c200 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:56.202089 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9ea4c36-c2a8-4cc8-9b95-b35eada0c200 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:56.202567 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9ea4c36-c2a8-4cc8-9b95-b35eada0c200 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:56.363924 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-b9ea4c36-c2a8-4cc8-9b95-b35eada0c200 None None] MFA Rules not processed for user `704501227d4c4de9821fb2f427aaffad`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:33:56.367520 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-b9ea4c36-c2a8-4cc8-9b95-b35eada0c200 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:33:56.443446 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47326410-001d-4812-b78e-305db0dc3bd9 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:56.635602 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:33:56.656964 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:33:56.720082 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 851/1703] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:33:56 2022] POST /identity/v3/auth/tokens => generated 3232 bytes in 517 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:33:56.724910 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94852961-4bc4-41fa-ad5c-54ea428b9190 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:56.725392 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94852961-4bc4-41fa-ad5c-54ea428b9190 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:56.725742 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94852961-4bc4-41fa-ad5c-54ea428b9190 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:56.727340 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac9180e5-3a44-477d-888f-5c860fbd06e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:56.728896 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac9180e5-3a44-477d-888f-5c860fbd06e5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:56.728896 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac9180e5-3a44-477d-888f-5c860fbd06e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:56.728896 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac9180e5-3a44-477d-888f-5c860fbd06e5 admin admin] PATH_INFO: `/v3/projects/65263321fac44ae69c91924f4bee07d1/users/3f6b1a97c07b4090bca1fb483f32a4f6/roles/a6e562cdc96c4be691d638990f8403fd` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:56.831234 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac9180e5-3a44-477d-888f-5c860fbd06e5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=65263321fac44ae69c91924f4bee07d1, user_id=3f6b1a97c07b4090bca1fb483f32a4f6, role_id=a6e562cdc96c4be691d638990f8403fd)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:56.831322 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-94852961-4bc4-41fa-ad5c-54ea428b9190 None None] MFA Rules not processed for user `71b71ed8570941f9946baad3cc7e7fba`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:33:56.834228 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-94852961-4bc4-41fa-ad5c-54ea428b9190 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:33:56.844482 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac9180e5-3a44-477d-888f-5c860fbd06e5 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:56.888504 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 853/1704] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:33:56 2022] PUT /identity/v3/projects/65263321fac44ae69c91924f4bee07d1/users/3f6b1a97c07b4090bca1fb483f32a4f6/roles/a6e562cdc96c4be691d638990f8403fd => generated 0 bytes in 727 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:33:56.890293 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a96f6660-dae7-40d7-9387-21b3db1103f2 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:56.891139 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a96f6660-dae7-40d7-9387-21b3db1103f2 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:56.894743 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a96f6660-dae7-40d7-9387-21b3db1103f2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:57.098972 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-a96f6660-dae7-40d7-9387-21b3db1103f2 None None] MFA Rules not processed for user `0c0b0da87b124de1be00faec4bbdb404`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:33:57.100301 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-a96f6660-dae7-40d7-9387-21b3db1103f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:33:57.114931 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 852/1705] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:33:56 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 391 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:33:57.117241 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94852961-4bc4-41fa-ad5c-54ea428b9190 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:57.192476 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94852961-4bc4-41fa-ad5c-54ea428b9190 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:57.284469 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 854/1706] 10.222.0.90 () {58 vars in 1009 bytes} [Mon Aug 29 21:33:56 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 395 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:33:57.285997 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a96f6660-dae7-40d7-9387-21b3db1103f2 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:57.446756 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c2cddc0-1eb2-47c1-9e9b-b18314fda5bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:57.449350 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c2cddc0-1eb2-47c1-9e9b-b18314fda5bd admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:57.449759 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c2cddc0-1eb2-47c1-9e9b-b18314fda5bd admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:57.450307 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c2cddc0-1eb2-47c1-9e9b-b18314fda5bd admin admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:57.452618 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c2cddc0-1eb2-47c1-9e9b-b18314fda5bd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:57.459431 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c2cddc0-1eb2-47c1-9e9b-b18314fda5bd admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:57.534379 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 853/1707] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:33:57 2022] POST /identity/v3/projects => generated 357 bytes in 422 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:33:57.537880 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c2cddc0-1eb2-47c1-9e9b-b18314fda5bd admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:57.540761 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-1c2cddc0-1eb2-47c1-9e9b-b18314fda5bd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:33:57.546462 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a96f6660-dae7-40d7-9387-21b3db1103f2 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:57.726831 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:33:57.738451 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:33:57.750219 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c2cddc0-1eb2-47c1-9e9b-b18314fda5bd admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:57.766933 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a39a03a-8cf9-4c25-9765-ba54a75f198d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a35b5b7dea7b421da056f063d2436ffb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:57.768893 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a39a03a-8cf9-4c25-9765-ba54a75f198d service nova] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:57.769179 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a39a03a-8cf9-4c25-9765-ba54a75f198d service nova] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:57.769479 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a39a03a-8cf9-4c25-9765-ba54a75f198d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:57.780906 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2b477df-4d84-49e1-94b7-31c19c373ddc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:57.781669 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2b477df-4d84-49e1-94b7-31c19c373ddc admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:57.783328 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2b477df-4d84-49e1-94b7-31c19c373ddc admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:57.783642 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2b477df-4d84-49e1-94b7-31c19c373ddc admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:57.784505 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2b477df-4d84-49e1-94b7-31c19c373ddc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:57.790679 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2b477df-4d84-49e1-94b7-31c19c373ddc admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:57.814556 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 855/1708] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:33:57 2022] GET /identity/v3/roles => generated 1420 bytes in 529 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:33:57.815516 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2b477df-4d84-49e1-94b7-31c19c373ddc admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:57.869008 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2b477df-4d84-49e1-94b7-31c19c373ddc admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:57.900729 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-878978a7-58e7-4920-bb55-ea9f7e6624ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:57.901370 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-878978a7-58e7-4920-bb55-ea9f7e6624ab admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:57.902021 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-878978a7-58e7-4920-bb55-ea9f7e6624ab admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:57.902021 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-878978a7-58e7-4920-bb55-ea9f7e6624ab admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:57.902622 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-878978a7-58e7-4920-bb55-ea9f7e6624ab admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:57.904597 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-878978a7-58e7-4920-bb55-ea9f7e6624ab admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:57.925546 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a39a03a-8cf9-4c25-9765-ba54a75f198d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:57.927529 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a39a03a-8cf9-4c25-9765-ba54a75f198d service nova] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:57.964280 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 854/1709] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:33:57 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 429 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:33:57.979589 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed13274d-9ab5-4d15-9593-48b93c4afdd3 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:57.980976 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed13274d-9ab5-4d15-9593-48b93c4afdd3 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:57.982509 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed13274d-9ab5-4d15-9593-48b93c4afdd3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:57.984664 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 856/1710] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:33:57 2022] POST /identity/v3/users => generated 285 bytes in 170 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:33:57.987969 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-878978a7-58e7-4920-bb55-ea9f7e6624ab admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:57.987969 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-878978a7-58e7-4920-bb55-ea9f7e6624ab admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:33:58.070057 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-ed13274d-9ab5-4d15-9593-48b93c4afdd3 None None] MFA Rules not processed for user `96d3a0e28dc247b5a7ee9434a4d6554c`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:33:58.071970 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-ed13274d-9ab5-4d15-9593-48b93c4afdd3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:33:58.255439 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 855/1711] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:33:57 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 291 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:33:58.262018 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed13274d-9ab5-4d15-9593-48b93c4afdd3 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:58.300683 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-878978a7-58e7-4920-bb55-ea9f7e6624ab admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:58.302765 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed13274d-9ab5-4d15-9593-48b93c4afdd3 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:58.313143 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d22a0c4-a35f-4776-b3a8-262741b81945 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:58.316047 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d22a0c4-a35f-4776-b3a8-262741b81945 admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:58.316754 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d22a0c4-a35f-4776-b3a8-262741b81945 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:58.317719 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d22a0c4-a35f-4776-b3a8-262741b81945 admin admin] PATH_INFO: `/v3/projects/65263321fac44ae69c91924f4bee07d1/users/3f6b1a97c07b4090bca1fb483f32a4f6/roles/a6e562cdc96c4be691d638990f8403fd` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:58.320937 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e7eeb72-edb2-4cba-8058-28206232d695 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '139bbbde6d884b42b5440727926e2c58', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:58.321452 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d22a0c4-a35f-4776-b3a8-262741b81945 admin admin] RBAC: Authorizing `identity:create_grant(project_id=65263321fac44ae69c91924f4bee07d1, user_id=3f6b1a97c07b4090bca1fb483f32a4f6, role_id=a6e562cdc96c4be691d638990f8403fd)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:58.322383 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e7eeb72-edb2-4cba-8058-28206232d695 service swift] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:58.322682 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e7eeb72-edb2-4cba-8058-28206232d695 service swift] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:58.323047 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e7eeb72-edb2-4cba-8058-28206232d695 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:58.328470 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d22a0c4-a35f-4776-b3a8-262741b81945 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:58.369776 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 856/1712] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:33:58 2022] PUT /identity/v3/projects/65263321fac44ae69c91924f4bee07d1/users/3f6b1a97c07b4090bca1fb483f32a4f6/roles/a6e562cdc96c4be691d638990f8403fd => generated 0 bytes in 114 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:33:58.393253 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d22a0c4-a35f-4776-b3a8-262741b81945 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:58.626690 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e7eeb72-edb2-4cba-8058-28206232d695 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:58.633123 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e7eeb72-edb2-4cba-8058-28206232d695 service swift] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:58.670402 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 857/1713] 10.222.0.90 () {60 vars in 1384 bytes} [Mon Aug 29 21:33:57 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 684 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 29 21:33:58.673850 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8424cedb-d24b-410d-87e4-18e0320550a3 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:58.674280 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8424cedb-d24b-410d-87e4-18e0320550a3 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:58.674568 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8424cedb-d24b-410d-87e4-18e0320550a3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:58.682854 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d22a0c4-a35f-4776-b3a8-262741b81945 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:58.730324 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe65e76d-3bc6-4e42-a3b9-7163b0e5b999 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:58.738610 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe65e76d-3bc6-4e42-a3b9-7163b0e5b999 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:58.740395 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe65e76d-3bc6-4e42-a3b9-7163b0e5b999 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:58.742749 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe65e76d-3bc6-4e42-a3b9-7163b0e5b999 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:58.745203 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe65e76d-3bc6-4e42-a3b9-7163b0e5b999 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:58.754782 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe65e76d-3bc6-4e42-a3b9-7163b0e5b999 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:58.829718 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-8424cedb-d24b-410d-87e4-18e0320550a3 None None] MFA Rules not processed for user `96d3a0e28dc247b5a7ee9434a4d6554c`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:33:58.838622 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-8424cedb-d24b-410d-87e4-18e0320550a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:33:58.864227 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 857/1714] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:33:58 2022] GET /identity/v3/roles => generated 1420 bytes in 494 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:33:58.872945 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d292249a-38a3-4613-9894-767a40515836 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:58.877514 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d292249a-38a3-4613-9894-767a40515836 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:58.878540 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d292249a-38a3-4613-9894-767a40515836 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:59.027727 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-d292249a-38a3-4613-9894-767a40515836 None None] MFA Rules not processed for user `3f6b1a97c07b4090bca1fb483f32a4f6`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:33:59.029498 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-d292249a-38a3-4613-9894-767a40515836 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:33:59.036954 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:33:59.218275 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:33:59.256552 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 858/1715] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:33:58 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 586 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:33:59.258312 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8424cedb-d24b-410d-87e4-18e0320550a3 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:59.290962 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8424cedb-d24b-410d-87e4-18e0320550a3 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:59.300908 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 858/1716] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:33:58 2022] POST /identity/v3/auth/tokens => generated 3353 bytes in 436 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:33:59.320495 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fbfe11c-5b4c-47fe-b30f-910e6e980efe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:59.321696 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fbfe11c-5b4c-47fe-b30f-910e6e980efe admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:59.321856 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d292249a-38a3-4613-9894-767a40515836 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:59.322341 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fbfe11c-5b4c-47fe-b30f-910e6e980efe admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:59.322816 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fbfe11c-5b4c-47fe-b30f-910e6e980efe admin admin] PATH_INFO: `/v3/projects/a24e2a26c6fe46f8a2b7b7306c62c42f/users/e540d72ab72f4a92923e780ec4574ff8/roles/a6e562cdc96c4be691d638990f8403fd` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:59.324062 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-d292249a-38a3-4613-9894-767a40515836 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:33:59.368623 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fbfe11c-5b4c-47fe-b30f-910e6e980efe admin admin] RBAC: Authorizing `identity:create_grant(project_id=a24e2a26c6fe46f8a2b7b7306c62c42f, user_id=e540d72ab72f4a92923e780ec4574ff8, role_id=a6e562cdc96c4be691d638990f8403fd)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:33:59.374417 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fbfe11c-5b4c-47fe-b30f-910e6e980efe admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:33:59.426132 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 859/1717] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:33:59 2022] PUT /identity/v3/projects/a24e2a26c6fe46f8a2b7b7306c62c42f/users/e540d72ab72f4a92923e780ec4574ff8/roles/a6e562cdc96c4be691d638990f8403fd => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:33:59.428240 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fbfe11c-5b4c-47fe-b30f-910e6e980efe admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:33:59.568295 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d292249a-38a3-4613-9894-767a40515836 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:59.767284 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fbfe11c-5b4c-47fe-b30f-910e6e980efe admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:33:59.786696 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75ee1efe-0e2d-406e-a3ef-6a7a79fd98f3 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a35b5b7dea7b421da056f063d2436ffb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:59.789742 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75ee1efe-0e2d-406e-a3ef-6a7a79fd98f3 service nova] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:59.792052 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75ee1efe-0e2d-406e-a3ef-6a7a79fd98f3 service nova] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:59.792343 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75ee1efe-0e2d-406e-a3ef-6a7a79fd98f3 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:59.807486 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f70aac32-3725-4567-97f2-bd044f9d4c31 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dc226f7ddcb49ecacb51ce57fdfc11b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:33:59.810450 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f70aac32-3725-4567-97f2-bd044f9d4c31 service cinder] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:33:59.810980 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f70aac32-3725-4567-97f2-bd044f9d4c31 service cinder] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:33:59.814304 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f70aac32-3725-4567-97f2-bd044f9d4c31 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:33:59.996448 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75ee1efe-0e2d-406e-a3ef-6a7a79fd98f3 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:00.001172 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75ee1efe-0e2d-406e-a3ef-6a7a79fd98f3 service nova] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:00.030292 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 860/1718] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:33:59 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 603 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:34:00.033188 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75ee1efe-0e2d-406e-a3ef-6a7a79fd98f3 service nova] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:00.051545 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f70aac32-3725-4567-97f2-bd044f9d4c31 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:00.058320 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f70aac32-3725-4567-97f2-bd044f9d4c31 service cinder] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:00.098370 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 859/1719] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:33:59 2022] GET /identity/v3/auth/tokens => generated 3353 bytes in 777 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 21:34:00.101867 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f70aac32-3725-4567-97f2-bd044f9d4c31 service cinder] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:00.106841 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-f70aac32-3725-4567-97f2-bd044f9d4c31 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:00.133689 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:34:00.328472 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75ee1efe-0e2d-406e-a3ef-6a7a79fd98f3 service nova] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:00.359099 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14bf1374-6be6-4da0-96cb-e9d797641f7d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:00.359099 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14bf1374-6be6-4da0-96cb-e9d797641f7d admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:00.359099 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14bf1374-6be6-4da0-96cb-e9d797641f7d admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:00.359449 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14bf1374-6be6-4da0-96cb-e9d797641f7d admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:00.360020 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14bf1374-6be6-4da0-96cb-e9d797641f7d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:00.367954 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14bf1374-6be6-4da0-96cb-e9d797641f7d admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:00.376936 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:34:00.427288 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f70aac32-3725-4567-97f2-bd044f9d4c31 service cinder] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:00.429484 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 861/1720] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:34:00 2022] GET /identity/v3/roles => generated 1420 bytes in 398 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:34:00.434591 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14bf1374-6be6-4da0-96cb-e9d797641f7d admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:00.455893 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11fa49ae-e933-49bd-b1cd-35a01e1fa40d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:00.456566 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11fa49ae-e933-49bd-b1cd-35a01e1fa40d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:00.456838 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11fa49ae-e933-49bd-b1cd-35a01e1fa40d admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:00.457110 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11fa49ae-e933-49bd-b1cd-35a01e1fa40d admin admin] PATH_INFO: `/v3/projects/cbaa820ba6ab47108ffc321f1011c3e6` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:00.466753 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11fa49ae-e933-49bd-b1cd-35a01e1fa40d admin admin] RBAC: Authorizing `identity:delete_project(project_id=cbaa820ba6ab47108ffc321f1011c3e6)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:00.478224 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11fa49ae-e933-49bd-b1cd-35a01e1fa40d admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:00.485598 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14bf1374-6be6-4da0-96cb-e9d797641f7d admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:00.505688 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4a5fbae-c65c-4ff3-9413-07429572ec63 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:00.509008 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4a5fbae-c65c-4ff3-9413-07429572ec63 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:00.509665 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4a5fbae-c65c-4ff3-9413-07429572ec63 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:00.510187 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4a5fbae-c65c-4ff3-9413-07429572ec63 admin admin] PATH_INFO: `/v3/projects/a24e2a26c6fe46f8a2b7b7306c62c42f/users/e540d72ab72f4a92923e780ec4574ff8/roles/a6e562cdc96c4be691d638990f8403fd` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:00.518329 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4a5fbae-c65c-4ff3-9413-07429572ec63 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a24e2a26c6fe46f8a2b7b7306c62c42f, user_id=e540d72ab72f4a92923e780ec4574ff8, role_id=a6e562cdc96c4be691d638990f8403fd)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:00.528249 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4a5fbae-c65c-4ff3-9413-07429572ec63 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:00.555697 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 862/1721] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:34:00 2022] PUT /identity/v3/projects/a24e2a26c6fe46f8a2b7b7306c62c42f/users/e540d72ab72f4a92923e780ec4574ff8/roles/a6e562cdc96c4be691d638990f8403fd => generated 0 bytes in 122 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:34:00.566315 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb24672e-ec82-43b2-ab40-a4ec6deaefdd None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:00.566958 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb24672e-ec82-43b2-ab40-a4ec6deaefdd None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:00.566958 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb24672e-ec82-43b2-ab40-a4ec6deaefdd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:00.640008 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-bb24672e-ec82-43b2-ab40-a4ec6deaefdd None None] MFA Rules not processed for user `e540d72ab72f4a92923e780ec4574ff8`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:34:00.641267 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-bb24672e-ec82-43b2-ab40-a4ec6deaefdd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:00.843678 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-11fa49ae-e933-49bd-b1cd-35a01e1fa40d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cbaa820ba6ab47108ffc321f1011c3e6', 'request_id': 'req-11fa49ae-e933-49bd-b1cd-35a01e1fa40d'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:00.863151 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 863/1722] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:34:00 2022] POST /identity/v3/auth/tokens => generated 3301 bytes in 294 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:34:00.863151 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb24672e-ec82-43b2-ab40-a4ec6deaefdd None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:00.863151 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-bb24672e-ec82-43b2-ab40-a4ec6deaefdd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:00.867337 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-11fa49ae-e933-49bd-b1cd-35a01e1fa40d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cbaa820ba6ab47108ffc321f1011c3e6', 'request_id': 'req-11fa49ae-e933-49bd-b1cd-35a01e1fa40d'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:00.885126 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-11fa49ae-e933-49bd-b1cd-35a01e1fa40d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cbaa820ba6ab47108ffc321f1011c3e6', 'request_id': 'req-11fa49ae-e933-49bd-b1cd-35a01e1fa40d'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:00.926632 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-11fa49ae-e933-49bd-b1cd-35a01e1fa40d admin admin] The token cache is being invalidate because project cbaa820ba6ab47108ffc321f1011c3e6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81434) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:34:00.927319 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-11fa49ae-e933-49bd-b1cd-35a01e1fa40d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:00.928357 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-11fa49ae-e933-49bd-b1cd-35a01e1fa40d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:00.937337 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 860/1723] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:34:00 2022] DELETE /identity/v3/projects/cbaa820ba6ab47108ffc321f1011c3e6 => generated 0 bytes in 835 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:34:00.941649 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11fa49ae-e933-49bd-b1cd-35a01e1fa40d admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:00.944840 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-11fa49ae-e933-49bd-b1cd-35a01e1fa40d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:01.028589 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb24672e-ec82-43b2-ab40-a4ec6deaefdd None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:01.148188 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11fa49ae-e933-49bd-b1cd-35a01e1fa40d admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:01.169927 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82909be8-9a16-4952-832e-42f8f9adec32 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dc226f7ddcb49ecacb51ce57fdfc11b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:01.170722 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82909be8-9a16-4952-832e-42f8f9adec32 service cinder] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:01.171039 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82909be8-9a16-4952-832e-42f8f9adec32 service cinder] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:01.171381 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82909be8-9a16-4952-832e-42f8f9adec32 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:01.185179 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-82909be8-9a16-4952-832e-42f8f9adec32 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:01.221304 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbf3c44a-5812-4664-8bf2-314baf18821d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dc226f7ddcb49ecacb51ce57fdfc11b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:01.222344 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbf3c44a-5812-4664-8bf2-314baf18821d service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:01.222857 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbf3c44a-5812-4664-8bf2-314baf18821d service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:01.223334 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbf3c44a-5812-4664-8bf2-314baf18821d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:01.227755 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-dbf3c44a-5812-4664-8bf2-314baf18821d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:01.343650 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:34:01.403193 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82909be8-9a16-4952-832e-42f8f9adec32 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:01.404852 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82909be8-9a16-4952-832e-42f8f9adec32 service cinder] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:01.432388 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 861/1724] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:34:00 2022] GET /identity/v3/auth/tokens => generated 3301 bytes in 495 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:34:01.434652 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87bdf3f1-6088-4d25-8ee5-92eec5d5b8d3 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:01.438020 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87bdf3f1-6088-4d25-8ee5-92eec5d5b8d3 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:01.438431 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87bdf3f1-6088-4d25-8ee5-92eec5d5b8d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:01.447120 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:34:01.496750 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-87bdf3f1-6088-4d25-8ee5-92eec5d5b8d3 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:34:01.500128 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-87bdf3f1-6088-4d25-8ee5-92eec5d5b8d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:01.521832 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbf3c44a-5812-4664-8bf2-314baf18821d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:01.527117 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbf3c44a-5812-4664-8bf2-314baf18821d service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:01.563470 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 864/1725] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:34:00 2022] GET /identity/v3/auth/tokens => generated 3286 bytes in 703 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:34:01.691703 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 862/1726] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:34:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 258 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:34:01.701715 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-718edc41-38bd-4590-8a39-893dfbd11c9a None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:01.702332 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-718edc41-38bd-4590-8a39-893dfbd11c9a None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:01.702622 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-718edc41-38bd-4590-8a39-893dfbd11c9a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:01.769544 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-718edc41-38bd-4590-8a39-893dfbd11c9a None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:34:01.778872 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-718edc41-38bd-4590-8a39-893dfbd11c9a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:01.867260 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87bdf3f1-6088-4d25-8ee5-92eec5d5b8d3 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:01.867708 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 865/1727] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:34:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:34:01.894270 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87bdf3f1-6088-4d25-8ee5-92eec5d5b8d3 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:01.937308 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9316c356-fc0d-492c-b976-5819ac5fd6b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:01.940354 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9316c356-fc0d-492c-b976-5819ac5fd6b0 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:01.940909 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9316c356-fc0d-492c-b976-5819ac5fd6b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:01.943501 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9316c356-fc0d-492c-b976-5819ac5fd6b0 admin admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:01.946703 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9316c356-fc0d-492c-b976-5819ac5fd6b0 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:34:01.947581 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9316c356-fc0d-492c-b976-5819ac5fd6b0 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:01.955877 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9316c356-fc0d-492c-b976-5819ac5fd6b0 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:02.042671 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 863/1728] 10.222.0.90 () {58 vars in 1153 bytes} [Mon Aug 29 21:34:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:34:02.047576 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-718edc41-38bd-4590-8a39-893dfbd11c9a None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:02.073385 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9316c356-fc0d-492c-b976-5819ac5fd6b0 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:02.083125 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-718edc41-38bd-4590-8a39-893dfbd11c9a None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:02.101346 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9316c356-fc0d-492c-b976-5819ac5fd6b0 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:02.110213 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-059ae071-eaee-440a-9ed4-7dd87c6aa0e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:02.118970 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-059ae071-eaee-440a-9ed4-7dd87c6aa0e4 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:02.119755 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-059ae071-eaee-440a-9ed4-7dd87c6aa0e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:02.120233 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-059ae071-eaee-440a-9ed4-7dd87c6aa0e4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:02.121282 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-059ae071-eaee-440a-9ed4-7dd87c6aa0e4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:02.124539 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-059ae071-eaee-440a-9ed4-7dd87c6aa0e4 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:02.128757 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aee1cfaf-e803-4efe-aa33-c4fedeb824a6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dc226f7ddcb49ecacb51ce57fdfc11b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:02.129592 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aee1cfaf-e803-4efe-aa33-c4fedeb824a6 service cinder] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:02.129948 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aee1cfaf-e803-4efe-aa33-c4fedeb824a6 service cinder] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:02.130286 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aee1cfaf-e803-4efe-aa33-c4fedeb824a6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:02.145892 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-aee1cfaf-e803-4efe-aa33-c4fedeb824a6 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:02.245739 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 866/1729] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:34:02 2022] POST /identity/v3/projects => generated 371 bytes in 199 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:34:02.254997 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eea7957-c3ee-4028-aa9f-c169fa7bf034 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:02.263161 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eea7957-c3ee-4028-aa9f-c169fa7bf034 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:02.265179 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eea7957-c3ee-4028-aa9f-c169fa7bf034 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:02.275733 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 867/1730] 10.222.0.90 () {54 vars in 912 bytes} [Mon Aug 29 21:34:02 2022] GET /identity => generated 270 bytes in 29 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:34:02.280114 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eea7957-c3ee-4028-aa9f-c169fa7bf034 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:02.378315 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aee1cfaf-e803-4efe-aa33-c4fedeb824a6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:02.381823 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aee1cfaf-e803-4efe-aa33-c4fedeb824a6 service cinder] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:02.416317 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 864/1731] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:34:02 2022] GET /identity/v3/auth/tokens => generated 3232 bytes in 344 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:34:02.419467 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b493fd1-f5db-426e-b992-617a091a74a0 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:02.419811 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b493fd1-f5db-426e-b992-617a091a74a0 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:02.420088 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b493fd1-f5db-426e-b992-617a091a74a0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:02.465525 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-1b493fd1-f5db-426e-b992-617a091a74a0 None None] MFA Rules not processed for user `0e69b7f0c3574abc9de5fb580d22152a`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:34:02.466959 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-1b493fd1-f5db-426e-b992-617a091a74a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:02.592077 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eea7957-c3ee-4028-aa9f-c169fa7bf034 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:02.608160 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a36bfec4-507e-4b47-92b5-641c934b8971 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:02.609319 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a36bfec4-507e-4b47-92b5-641c934b8971 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:02.610944 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a36bfec4-507e-4b47-92b5-641c934b8971 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:02.611278 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a36bfec4-507e-4b47-92b5-641c934b8971 admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:02.614282 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a36bfec4-507e-4b47-92b5-641c934b8971 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:02.628926 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a36bfec4-507e-4b47-92b5-641c934b8971 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:02.733995 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 865/1732] 10.222.0.90 () {58 vars in 1009 bytes} [Mon Aug 29 21:34:02 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 313 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:34:02.734218 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 868/1733] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:34:02 2022] POST /identity/v3/users => generated 286 bytes in 454 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:34:02.736355 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a36bfec4-507e-4b47-92b5-641c934b8971 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:02.773412 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a36bfec4-507e-4b47-92b5-641c934b8971 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:02.798964 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67b25f0f-665c-4fc5-8be8-e14c68cf4a38 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:02.799877 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67b25f0f-665c-4fc5-8be8-e14c68cf4a38 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:02.800555 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67b25f0f-665c-4fc5-8be8-e14c68cf4a38 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:02.801828 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67b25f0f-665c-4fc5-8be8-e14c68cf4a38 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:02.803829 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67b25f0f-665c-4fc5-8be8-e14c68cf4a38 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:02.830682 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b493fd1-f5db-426e-b992-617a091a74a0 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:02.832389 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-1b493fd1-f5db-426e-b992-617a091a74a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:02.833210 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67b25f0f-665c-4fc5-8be8-e14c68cf4a38 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:02.869606 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 869/1734] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:34:02 2022] GET /identity/v3/roles => generated 1420 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:34:02.872582 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b493fd1-f5db-426e-b992-617a091a74a0 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:02.875086 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67b25f0f-665c-4fc5-8be8-e14c68cf4a38 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:02.887879 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0256b3e-a357-4260-94f3-146d5eb53680 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:02.888939 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0256b3e-a357-4260-94f3-146d5eb53680 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:02.889456 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0256b3e-a357-4260-94f3-146d5eb53680 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:02.889977 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0256b3e-a357-4260-94f3-146d5eb53680 admin admin] PATH_INFO: `/v3/users/71b71ed8570941f9946baad3cc7e7fba` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:02.896304 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0256b3e-a357-4260-94f3-146d5eb53680 admin admin] RBAC: Authorizing `identity:delete_user(user_id=71b71ed8570941f9946baad3cc7e7fba)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:02.906633 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0256b3e-a357-4260-94f3-146d5eb53680 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:02.925656 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67b25f0f-665c-4fc5-8be8-e14c68cf4a38 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:02.963961 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b33d3d7c-0a34-4027-b3d5-f0de7925ef38 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:02.964716 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b33d3d7c-0a34-4027-b3d5-f0de7925ef38 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:02.965086 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b33d3d7c-0a34-4027-b3d5-f0de7925ef38 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:02.965431 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b33d3d7c-0a34-4027-b3d5-f0de7925ef38 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:02.966312 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b33d3d7c-0a34-4027-b3d5-f0de7925ef38 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:02.973330 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b33d3d7c-0a34-4027-b3d5-f0de7925ef38 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:03.000987 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=99,keystone:DELETE=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:34:03.023833 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 870/1735] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:34:02 2022] GET /identity/v3/roles => generated 1420 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 29 21:34:03.028037 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b33d3d7c-0a34-4027-b3d5-f0de7925ef38 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:03.098166 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b33d3d7c-0a34-4027-b3d5-f0de7925ef38 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:03.139783 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-f0256b3e-a357-4260-94f3-146d5eb53680 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '71b71ed8570941f9946baad3cc7e7fba'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:03.146404 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fb531e6-c442-4179-b416-c8c05612ea98 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:03.148024 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb531e6-c442-4179-b416-c8c05612ea98 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:03.149283 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb531e6-c442-4179-b416-c8c05612ea98 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:03.149730 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb531e6-c442-4179-b416-c8c05612ea98 admin admin] PATH_INFO: `/v3/projects/ea55bac9f2e7482fb9badfcd17e05741/users/da585500ed9943738914e64f14f4d53b/roles/02e242d5520740d4b77a5f4008c7eb57` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:03.174880 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fb531e6-c442-4179-b416-c8c05612ea98 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ea55bac9f2e7482fb9badfcd17e05741, user_id=da585500ed9943738914e64f14f4d53b, role_id=02e242d5520740d4b77a5f4008c7eb57)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:03.178415 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fb531e6-c442-4179-b416-c8c05612ea98 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:03.179192 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-f0256b3e-a357-4260-94f3-146d5eb53680 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '71b71ed8570941f9946baad3cc7e7fba'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:03.202037 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-f0256b3e-a357-4260-94f3-146d5eb53680 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '71b71ed8570941f9946baad3cc7e7fba'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:03.202889 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-f0256b3e-a357-4260-94f3-146d5eb53680 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '71b71ed8570941f9946baad3cc7e7fba'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:03.203714 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-f0256b3e-a357-4260-94f3-146d5eb53680 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '71b71ed8570941f9946baad3cc7e7fba'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:03.218188 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 871/1736] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:34:03 2022] PUT /identity/v3/projects/ea55bac9f2e7482fb9badfcd17e05741/users/da585500ed9943738914e64f14f4d53b/roles/02e242d5520740d4b77a5f4008c7eb57 => generated 0 bytes in 190 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:34:03.249027 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee05acf7-8084-4479-87d2-1cff6a600d00 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:03.249640 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee05acf7-8084-4479-87d2-1cff6a600d00 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:03.251191 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee05acf7-8084-4479-87d2-1cff6a600d00 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:03.296983 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 866/1737] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:34:02 2022] DELETE /identity/v3/users/71b71ed8570941f9946baad3cc7e7fba => generated 0 bytes in 467 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:34:03.304774 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b975ea8-52e1-4951-ba21-b7e21a2385e8 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:03.312225 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b975ea8-52e1-4951-ba21-b7e21a2385e8 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:03.312720 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b975ea8-52e1-4951-ba21-b7e21a2385e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:03.392837 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-6b975ea8-52e1-4951-ba21-b7e21a2385e8 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:34:03.394254 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-6b975ea8-52e1-4951-ba21-b7e21a2385e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:03.415722 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-ee05acf7-8084-4479-87d2-1cff6a600d00 None None] MFA Rules not processed for user `da585500ed9943738914e64f14f4d53b`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:34:03.418539 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-ee05acf7-8084-4479-87d2-1cff6a600d00 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:03.627708 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:34:03.674119 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 867/1738] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:34:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 373 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:34:03.676568 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b975ea8-52e1-4951-ba21-b7e21a2385e8 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:03.695814 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-6b975ea8-52e1-4951-ba21-b7e21a2385e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:03.711514 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 872/1739] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:34:03 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 470 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:34:03.712734 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee05acf7-8084-4479-87d2-1cff6a600d00 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:03.718382 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-ee05acf7-8084-4479-87d2-1cff6a600d00 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:03.957979 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee05acf7-8084-4479-87d2-1cff6a600d00 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:03.966663 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fe09c59-8f4b-48f1-9db9-0dc506afe9d2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:03.967514 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fe09c59-8f4b-48f1-9db9-0dc506afe9d2 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:03.967903 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fe09c59-8f4b-48f1-9db9-0dc506afe9d2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:03.968281 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fe09c59-8f4b-48f1-9db9-0dc506afe9d2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:03.979953 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b975ea8-52e1-4951-ba21-b7e21a2385e8 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:03.985932 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94e91d07-28ba-4153-bda7-d9b5cfe29291 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:03.986886 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94e91d07-28ba-4153-bda7-d9b5cfe29291 service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:03.987455 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94e91d07-28ba-4153-bda7-d9b5cfe29291 service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:03.988024 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94e91d07-28ba-4153-bda7-d9b5cfe29291 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:03.990665 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-94e91d07-28ba-4153-bda7-d9b5cfe29291 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:03.998282 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fe09c59-8f4b-48f1-9db9-0dc506afe9d2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:03.999900 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fe09c59-8f4b-48f1-9db9-0dc506afe9d2 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:04.029388 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94e91d07-28ba-4153-bda7-d9b5cfe29291 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:04.035180 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 873/1740] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:34:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 323 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:34:04.035808 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94e91d07-28ba-4153-bda7-d9b5cfe29291 service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:04.042789 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fe09c59-8f4b-48f1-9db9-0dc506afe9d2 service neutron] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:04.051656 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-5fe09c59-8f4b-48f1-9db9-0dc506afe9d2 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:04.069002 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 868/1741] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:34:03 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 393 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:34:04.097377 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fe09c59-8f4b-48f1-9db9-0dc506afe9d2 service neutron] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:04.144491 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-002c89ec-87bd-4adf-88e5-a34f0ae1ee82 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:04.145570 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-002c89ec-87bd-4adf-88e5-a34f0ae1ee82 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:04.146099 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-002c89ec-87bd-4adf-88e5-a34f0ae1ee82 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:04.146864 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-002c89ec-87bd-4adf-88e5-a34f0ae1ee82 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:04.148357 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-002c89ec-87bd-4adf-88e5-a34f0ae1ee82 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:04.152052 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-002c89ec-87bd-4adf-88e5-a34f0ae1ee82 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:04.246526 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 874/1742] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:34:04 2022] POST /identity/v3/projects => generated 371 bytes in 211 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:34:04.247898 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94e91d07-28ba-4153-bda7-d9b5cfe29291 service neutron] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:04.276886 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94e91d07-28ba-4153-bda7-d9b5cfe29291 service neutron] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:04.490714 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:DELETE=9,keystone:SELECT=90,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:34:04.634371 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74cb34e3-36f3-46af-84a1-d17cb61520d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:04.635497 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74cb34e3-36f3-46af-84a1-d17cb61520d4 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:04.635790 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74cb34e3-36f3-46af-84a1-d17cb61520d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:04.636066 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74cb34e3-36f3-46af-84a1-d17cb61520d4 admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:04.636738 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74cb34e3-36f3-46af-84a1-d17cb61520d4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:04.638776 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74cb34e3-36f3-46af-84a1-d17cb61520d4 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:04.775487 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 869/1743] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:34:04 2022] POST /identity/v3/users => generated 292 bytes in 528 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:34:04.781949 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-002c89ec-87bd-4adf-88e5-a34f0ae1ee82 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:04.811164 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-002c89ec-87bd-4adf-88e5-a34f0ae1ee82 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:04.841821 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c57a3353-a34c-4490-91a6-fc94594de21c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:04.842674 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c57a3353-a34c-4490-91a6-fc94594de21c admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:04.853065 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c57a3353-a34c-4490-91a6-fc94594de21c admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:04.854287 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c57a3353-a34c-4490-91a6-fc94594de21c admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:04.856936 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c57a3353-a34c-4490-91a6-fc94594de21c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:04.865008 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c57a3353-a34c-4490-91a6-fc94594de21c admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:04.931622 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 875/1744] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:34:04 2022] GET /identity/v3/roles => generated 1420 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:34:04.932876 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74cb34e3-36f3-46af-84a1-d17cb61520d4 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:04.968929 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74cb34e3-36f3-46af-84a1-d17cb61520d4 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:05.020774 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87152a4c-36d9-46ac-8643-ccae6ff5ae5e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:05.023435 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87152a4c-36d9-46ac-8643-ccae6ff5ae5e admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:05.025927 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87152a4c-36d9-46ac-8643-ccae6ff5ae5e admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:05.026494 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87152a4c-36d9-46ac-8643-ccae6ff5ae5e admin admin] PATH_INFO: `/v3/projects/c64b966440f5415bb9d75cb47ac1ecae/users/2dcadec16df6466799d0ffc9da063109/roles/a6e562cdc96c4be691d638990f8403fd` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:05.123399 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87152a4c-36d9-46ac-8643-ccae6ff5ae5e admin admin] RBAC: Authorizing `identity:create_grant(project_id=c64b966440f5415bb9d75cb47ac1ecae, user_id=2dcadec16df6466799d0ffc9da063109, role_id=a6e562cdc96c4be691d638990f8403fd)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:05.128823 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87152a4c-36d9-46ac-8643-ccae6ff5ae5e admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:05.201205 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 870/1745] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:34:04 2022] PUT /identity/v3/projects/c64b966440f5415bb9d75cb47ac1ecae/users/2dcadec16df6466799d0ffc9da063109/roles/a6e562cdc96c4be691d638990f8403fd => generated 0 bytes in 268 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:34:05.205579 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c57a3353-a34c-4490-91a6-fc94594de21c admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:05.246014 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c57a3353-a34c-4490-91a6-fc94594de21c admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:05.388805 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87152a4c-36d9-46ac-8643-ccae6ff5ae5e admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:05.391386 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-87152a4c-36d9-46ac-8643-ccae6ff5ae5e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:05.452274 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-044863ff-27af-42dc-aba3-8e0e91830043 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:05.453895 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-044863ff-27af-42dc-aba3-8e0e91830043 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:05.454200 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-044863ff-27af-42dc-aba3-8e0e91830043 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:05.454462 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-044863ff-27af-42dc-aba3-8e0e91830043 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:05.455108 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-044863ff-27af-42dc-aba3-8e0e91830043 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:05.464226 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-044863ff-27af-42dc-aba3-8e0e91830043 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:05.514571 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 876/1746] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:34:05 2022] GET /identity/v3/roles => generated 1420 bytes in 309 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:34:05.516480 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-044863ff-27af-42dc-aba3-8e0e91830043 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:05.570630 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-044863ff-27af-42dc-aba3-8e0e91830043 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:05.592168 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cd6cbf2-bb5d-4a94-ad8c-af87f299ab31 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:05.592949 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cd6cbf2-bb5d-4a94-ad8c-af87f299ab31 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:05.593231 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cd6cbf2-bb5d-4a94-ad8c-af87f299ab31 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:05.593543 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cd6cbf2-bb5d-4a94-ad8c-af87f299ab31 admin admin] PATH_INFO: `/v3/projects/c64b966440f5415bb9d75cb47ac1ecae/users/2dcadec16df6466799d0ffc9da063109/roles/a6e562cdc96c4be691d638990f8403fd` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:05.603553 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cd6cbf2-bb5d-4a94-ad8c-af87f299ab31 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c64b966440f5415bb9d75cb47ac1ecae, user_id=2dcadec16df6466799d0ffc9da063109, role_id=a6e562cdc96c4be691d638990f8403fd)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:05.616762 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cd6cbf2-bb5d-4a94-ad8c-af87f299ab31 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:05.664239 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87152a4c-36d9-46ac-8643-ccae6ff5ae5e admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:05.678278 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 877/1747] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:34:05 2022] PUT /identity/v3/projects/c64b966440f5415bb9d75cb47ac1ecae/users/2dcadec16df6466799d0ffc9da063109/roles/a6e562cdc96c4be691d638990f8403fd => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:34:05.699690 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f134523-551b-4921-b1bf-b841cda31f4b None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:05.700106 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f134523-551b-4921-b1bf-b841cda31f4b None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:05.700350 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f134523-551b-4921-b1bf-b841cda31f4b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:05.813326 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-0f134523-551b-4921-b1bf-b841cda31f4b None None] MFA Rules not processed for user `2dcadec16df6466799d0ffc9da063109`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:34:05.817492 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-0f134523-551b-4921-b1bf-b841cda31f4b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:05.900755 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5c3f01e-0225-4817-815a-198cc0117bc7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:05.902132 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5c3f01e-0225-4817-815a-198cc0117bc7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:05.902642 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5c3f01e-0225-4817-815a-198cc0117bc7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:05.903128 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5c3f01e-0225-4817-815a-198cc0117bc7 admin admin] PATH_INFO: `/v3/projects/4ce47d12b3aa4d618ae4880856c5093f` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:05.908979 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5c3f01e-0225-4817-815a-198cc0117bc7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4ce47d12b3aa4d618ae4880856c5093f)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:05.912936 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5c3f01e-0225-4817-815a-198cc0117bc7 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:05.915116 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:34:06.090538 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-b5c3f01e-0225-4817-815a-198cc0117bc7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4ce47d12b3aa4d618ae4880856c5093f', 'request_id': 'req-b5c3f01e-0225-4817-815a-198cc0117bc7'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:06.094269 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=91,keystone:INSERT=4,keystone:DELETE=5 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:34:06.105616 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-b5c3f01e-0225-4817-815a-198cc0117bc7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4ce47d12b3aa4d618ae4880856c5093f', 'request_id': 'req-b5c3f01e-0225-4817-815a-198cc0117bc7'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:06.131137 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-b5c3f01e-0225-4817-815a-198cc0117bc7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4ce47d12b3aa4d618ae4880856c5093f', 'request_id': 'req-b5c3f01e-0225-4817-815a-198cc0117bc7'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:06.170225 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 878/1748] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:34:05 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 477 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:34:06.181056 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-b5c3f01e-0225-4817-815a-198cc0117bc7 admin admin] The token cache is being invalidate because project 4ce47d12b3aa4d618ae4880856c5093f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81434) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:34:06.185943 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-b5c3f01e-0225-4817-815a-198cc0117bc7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:06.187788 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-b5c3f01e-0225-4817-815a-198cc0117bc7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:06.190115 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f134523-551b-4921-b1bf-b841cda31f4b None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:06.192177 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-0f134523-551b-4921-b1bf-b841cda31f4b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:06.192660 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 871/1749] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:34:05 2022] DELETE /identity/v3/projects/4ce47d12b3aa4d618ae4880856c5093f => generated 0 bytes in 805 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 21:34:06.203986 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4143a3c3-3c54-4cac-8c91-38d56d19d1e8 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:06.204483 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4143a3c3-3c54-4cac-8c91-38d56d19d1e8 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:06.210599 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4143a3c3-3c54-4cac-8c91-38d56d19d1e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:06.300541 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-4143a3c3-3c54-4cac-8c91-38d56d19d1e8 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:34:06.301884 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-4143a3c3-3c54-4cac-8c91-38d56d19d1e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:06.354190 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f134523-551b-4921-b1bf-b841cda31f4b None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:06.519280 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 872/1750] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:34:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 309 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:34:06.524593 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4435ffb-479c-4027-81cb-b37d6bea42c7 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:06.525041 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4435ffb-479c-4027-81cb-b37d6bea42c7 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:06.525405 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4435ffb-479c-4027-81cb-b37d6bea42c7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:06.642497 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-e4435ffb-479c-4027-81cb-b37d6bea42c7 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:34:06.644497 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-e4435ffb-479c-4027-81cb-b37d6bea42c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:06.692581 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44093719-6ae9-4fb2-bcfa-bb90abbd7693 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dc226f7ddcb49ecacb51ce57fdfc11b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:06.693169 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44093719-6ae9-4fb2-bcfa-bb90abbd7693 service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:06.693453 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44093719-6ae9-4fb2-bcfa-bb90abbd7693 service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:06.693736 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44093719-6ae9-4fb2-bcfa-bb90abbd7693 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:06.703147 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-44093719-6ae9-4fb2-bcfa-bb90abbd7693 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:06.708400 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 873/1751] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:34:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 21:34:06.716657 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4435ffb-479c-4027-81cb-b37d6bea42c7 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:06.784448 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4435ffb-479c-4027-81cb-b37d6bea42c7 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:06.825499 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4068b29-fb38-4eeb-b1ed-f6cda0d6a99a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:06.838748 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4068b29-fb38-4eeb-b1ed-f6cda0d6a99a admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:06.839359 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4068b29-fb38-4eeb-b1ed-f6cda0d6a99a admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:06.856390 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4068b29-fb38-4eeb-b1ed-f6cda0d6a99a admin admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:06.860019 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4068b29-fb38-4eeb-b1ed-f6cda0d6a99a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:34:06.860340 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4068b29-fb38-4eeb-b1ed-f6cda0d6a99a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:06.873386 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4068b29-fb38-4eeb-b1ed-f6cda0d6a99a admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:06.941889 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 874/1752] 10.222.0.90 () {58 vars in 1153 bytes} [Mon Aug 29 21:34:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:34:06.941889 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4068b29-fb38-4eeb-b1ed-f6cda0d6a99a admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:06.996647 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4068b29-fb38-4eeb-b1ed-f6cda0d6a99a admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:07.022096 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4197b16-b2ba-452f-9d20-3bb503491bcd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:07.025871 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4197b16-b2ba-452f-9d20-3bb503491bcd admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:07.026129 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4197b16-b2ba-452f-9d20-3bb503491bcd admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:07.026292 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4197b16-b2ba-452f-9d20-3bb503491bcd admin admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:07.026981 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4197b16-b2ba-452f-9d20-3bb503491bcd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:07.036885 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4197b16-b2ba-452f-9d20-3bb503491bcd admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:07.187497 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 875/1753] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:34:06 2022] POST /identity/v3/projects => generated 405 bytes in 250 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:34:07.188345 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4197b16-b2ba-452f-9d20-3bb503491bcd admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:07.269442 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44093719-6ae9-4fb2-bcfa-bb90abbd7693 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:07.278016 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44093719-6ae9-4fb2-bcfa-bb90abbd7693 service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:07.363025 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 879/1754] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:34:06 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 1179 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:34:07.701832 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4197b16-b2ba-452f-9d20-3bb503491bcd admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:07.722423 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4549b3c-b0c9-4f19-a9cc-5e69e773be94 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:07.723938 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4549b3c-b0c9-4f19-a9cc-5e69e773be94 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:07.724392 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4549b3c-b0c9-4f19-a9cc-5e69e773be94 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:07.724798 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4549b3c-b0c9-4f19-a9cc-5e69e773be94 admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:07.731841 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4549b3c-b0c9-4f19-a9cc-5e69e773be94 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:07.738600 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4549b3c-b0c9-4f19-a9cc-5e69e773be94 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:07.801813 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 876/1755] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:34:07 2022] POST /identity/v3/users => generated 303 bytes in 614 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:34:07.808661 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44093719-6ae9-4fb2-bcfa-bb90abbd7693 service cinder] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:07.857138 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44093719-6ae9-4fb2-bcfa-bb90abbd7693 service cinder] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:07.898698 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-843758fe-dc2a-4e2a-aa02-68aeb854c4ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:07.908655 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-843758fe-dc2a-4e2a-aa02-68aeb854c4ce admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:07.914869 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-843758fe-dc2a-4e2a-aa02-68aeb854c4ce admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:07.915469 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-843758fe-dc2a-4e2a-aa02-68aeb854c4ce admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:07.917962 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-843758fe-dc2a-4e2a-aa02-68aeb854c4ce admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:07.921647 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-843758fe-dc2a-4e2a-aa02-68aeb854c4ce admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:08.061672 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 880/1756] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:34:07 2022] GET /identity/v3/roles => generated 1420 bytes in 254 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:34:08.067856 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4549b3c-b0c9-4f19-a9cc-5e69e773be94 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:08.103318 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4549b3c-b0c9-4f19-a9cc-5e69e773be94 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:08.169080 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc293a7e-fa67-49b3-8c72-1d856c33efe1 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:08.169587 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc293a7e-fa67-49b3-8c72-1d856c33efe1 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:08.169963 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc293a7e-fa67-49b3-8c72-1d856c33efe1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:08.188028 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5df11f71-0370-4547-88c4-8efabeace029 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:08.205142 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5df11f71-0370-4547-88c4-8efabeace029 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:08.206219 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5df11f71-0370-4547-88c4-8efabeace029 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:08.206219 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5df11f71-0370-4547-88c4-8efabeace029 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:08.207004 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5df11f71-0370-4547-88c4-8efabeace029 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:08.233354 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5df11f71-0370-4547-88c4-8efabeace029 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:08.274112 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-bc293a7e-fa67-49b3-8c72-1d856c33efe1 None None] MFA Rules not processed for user `0e69b7f0c3574abc9de5fb580d22152a`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:34:08.275659 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-bc293a7e-fa67-49b3-8c72-1d856c33efe1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:08.314366 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 877/1757] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:34:08 2022] GET /identity/v3/roles => generated 1420 bytes in 248 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:34:08.321668 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5df11f71-0370-4547-88c4-8efabeace029 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:08.371788 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5df11f71-0370-4547-88c4-8efabeace029 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:08.384041 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:34:08.401040 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ce480ad-f707-4c03-98f7-a345d435eb68 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:08.405469 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce480ad-f707-4c03-98f7-a345d435eb68 admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:08.408231 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce480ad-f707-4c03-98f7-a345d435eb68 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:08.408231 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce480ad-f707-4c03-98f7-a345d435eb68 admin admin] PATH_INFO: `/v3/projects/3287ead9dd7741ccaea942cc2027c7be/users/874d5f9ad9f84d7e8bb644a92e60d110/roles/02e242d5520740d4b77a5f4008c7eb57` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:08.598948 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 881/1758] 10.222.0.13 () {58 vars in 1015 bytes} [Mon Aug 29 21:34:08 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 431 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:34:08.600841 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12f7fa1d-6de6-4ae5-9925-632074b9a3eb None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:08.602813 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12f7fa1d-6de6-4ae5-9925-632074b9a3eb None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:08.605172 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12f7fa1d-6de6-4ae5-9925-632074b9a3eb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:08.637903 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ce480ad-f707-4c03-98f7-a345d435eb68 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3287ead9dd7741ccaea942cc2027c7be, user_id=874d5f9ad9f84d7e8bb644a92e60d110, role_id=02e242d5520740d4b77a5f4008c7eb57)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:08.664880 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ce480ad-f707-4c03-98f7-a345d435eb68 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:08.687816 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=93,keystone:DELETE=2,keystone:INSERT=5 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:34:08.763854 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-12f7fa1d-6de6-4ae5-9925-632074b9a3eb None None] MFA Rules not processed for user `bbbfb39899b04668ac1b7ff28d18e839`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:34:08.766023 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 878/1759] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:34:08 2022] PUT /identity/v3/projects/3287ead9dd7741ccaea942cc2027c7be/users/874d5f9ad9f84d7e8bb644a92e60d110/roles/02e242d5520740d4b77a5f4008c7eb57 => generated 0 bytes in 451 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:34:08.766952 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ce480ad-f707-4c03-98f7-a345d435eb68 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:08.769190 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-12f7fa1d-6de6-4ae5-9925-632074b9a3eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:08.788169 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-8ce480ad-f707-4c03-98f7-a345d435eb68 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:09.079317 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 882/1760] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:34:08 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 480 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:34:09.084282 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35626995-cccb-48f8-ac2a-97827bf0cabc None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:09.088701 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35626995-cccb-48f8-ac2a-97827bf0cabc None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:09.091184 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35626995-cccb-48f8-ac2a-97827bf0cabc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:09.122625 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ce480ad-f707-4c03-98f7-a345d435eb68 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:09.142332 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44bf68f4-3cc5-4a85-a299-66665c8a67ad service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:09.143145 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44bf68f4-3cc5-4a85-a299-66665c8a67ad service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:09.143536 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44bf68f4-3cc5-4a85-a299-66665c8a67ad service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:09.144272 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44bf68f4-3cc5-4a85-a299-66665c8a67ad service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:09.199385 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44bf68f4-3cc5-4a85-a299-66665c8a67ad service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:09.215808 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44bf68f4-3cc5-4a85-a299-66665c8a67ad service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:09.229428 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-35626995-cccb-48f8-ac2a-97827bf0cabc None None] MFA Rules not processed for user `874d5f9ad9f84d7e8bb644a92e60d110`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:34:09.231828 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-35626995-cccb-48f8-ac2a-97827bf0cabc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:09.287411 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 879/1761] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:34:08 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 521 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:34:09.291941 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44bf68f4-3cc5-4a85-a299-66665c8a67ad service neutron] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:09.305856 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-44bf68f4-3cc5-4a85-a299-66665c8a67ad service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:09.530558 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 883/1762] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:34:09 2022] POST /identity/v3/auth/tokens => generated 3282 bytes in 447 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:34:09.534407 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35626995-cccb-48f8-ac2a-97827bf0cabc None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:09.748604 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44bf68f4-3cc5-4a85-a299-66665c8a67ad service neutron] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:09.795201 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-104f1057-c722-4b9e-b286-f19b5cc2150c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:09.796083 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-104f1057-c722-4b9e-b286-f19b5cc2150c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:09.796597 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-104f1057-c722-4b9e-b286-f19b5cc2150c admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:09.812711 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-104f1057-c722-4b9e-b286-f19b5cc2150c admin admin] PATH_INFO: `/v3/users/da585500ed9943738914e64f14f4d53b` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:09.821356 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-104f1057-c722-4b9e-b286-f19b5cc2150c admin admin] RBAC: Authorizing `identity:delete_user(user_id=da585500ed9943738914e64f14f4d53b)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:09.849228 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-104f1057-c722-4b9e-b286-f19b5cc2150c admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:09.855523 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:34:09.950633 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35626995-cccb-48f8-ac2a-97827bf0cabc None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:09.995830 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56ddfe1d-4b50-4399-8f65-90cf105741a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:09.996780 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ddfe1d-4b50-4399-8f65-90cf105741a2 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:09.997444 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ddfe1d-4b50-4399-8f65-90cf105741a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:09.998014 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ddfe1d-4b50-4399-8f65-90cf105741a2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:09.999187 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56ddfe1d-4b50-4399-8f65-90cf105741a2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:10.001648 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56ddfe1d-4b50-4399-8f65-90cf105741a2 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:10.054942 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 884/1763] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:34:09 2022] POST /identity/v3/projects => generated 403 bytes in 523 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:34:10.056829 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56ddfe1d-4b50-4399-8f65-90cf105741a2 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:10.059802 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-56ddfe1d-4b50-4399-8f65-90cf105741a2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:10.173069 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56ddfe1d-4b50-4399-8f65-90cf105741a2 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:10.180562 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-104f1057-c722-4b9e-b286-f19b5cc2150c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'da585500ed9943738914e64f14f4d53b'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:10.181453 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54296d21-dd10-4f33-91da-cb53a73a96ab service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:10.182695 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54296d21-dd10-4f33-91da-cb53a73a96ab service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:10.183277 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54296d21-dd10-4f33-91da-cb53a73a96ab service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:10.183833 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54296d21-dd10-4f33-91da-cb53a73a96ab service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:10.215839 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-104f1057-c722-4b9e-b286-f19b5cc2150c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'da585500ed9943738914e64f14f4d53b'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:10.222576 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54296d21-dd10-4f33-91da-cb53a73a96ab service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:10.228032 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54296d21-dd10-4f33-91da-cb53a73a96ab service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:10.260067 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-104f1057-c722-4b9e-b286-f19b5cc2150c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'da585500ed9943738914e64f14f4d53b'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:10.261079 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-104f1057-c722-4b9e-b286-f19b5cc2150c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'da585500ed9943738914e64f14f4d53b'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:10.262458 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-104f1057-c722-4b9e-b286-f19b5cc2150c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'da585500ed9943738914e64f14f4d53b'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:10.282030 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 885/1764] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:34:10 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 227 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:34:10.294833 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54296d21-dd10-4f33-91da-cb53a73a96ab service neutron] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:10.297719 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-54296d21-dd10-4f33-91da-cb53a73a96ab service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:10.371104 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 880/1765] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:34:09 2022] DELETE /identity/v3/users/da585500ed9943738914e64f14f4d53b => generated 0 bytes in 1080 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:34:10.371752 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-104f1057-c722-4b9e-b286-f19b5cc2150c admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:10.386777 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-104f1057-c722-4b9e-b286-f19b5cc2150c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:10.480756 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:INSERT=2,keystone:SELECT=88,keystone:DELETE=10 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:34:10.595321 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54296d21-dd10-4f33-91da-cb53a73a96ab service neutron] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:10.750235 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-104f1057-c722-4b9e-b286-f19b5cc2150c admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:10.780488 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ea2ef89-c0ff-480d-80b2-63ed3dd5d188 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:10.784500 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ea2ef89-c0ff-480d-80b2-63ed3dd5d188 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:10.786155 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ea2ef89-c0ff-480d-80b2-63ed3dd5d188 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:10.788740 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ea2ef89-c0ff-480d-80b2-63ed3dd5d188 admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:10.789680 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ea2ef89-c0ff-480d-80b2-63ed3dd5d188 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:10.792046 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b82c70e0-4abe-476f-8f88-e11f87d0f8dd service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a35b5b7dea7b421da056f063d2436ffb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:10.793899 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b82c70e0-4abe-476f-8f88-e11f87d0f8dd service nova] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:10.795312 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b82c70e0-4abe-476f-8f88-e11f87d0f8dd service nova] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:10.796237 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ea2ef89-c0ff-480d-80b2-63ed3dd5d188 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:10.796674 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b82c70e0-4abe-476f-8f88-e11f87d0f8dd service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:10.807865 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-b82c70e0-4abe-476f-8f88-e11f87d0f8dd service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:10.891809 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 881/1766] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:34:10 2022] POST /identity/v3/users => generated 308 bytes in 520 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:34:10.893210 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c14584cd-6233-4a85-845a-670843c1d3c8 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:10.893495 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c14584cd-6233-4a85-845a-670843c1d3c8 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:10.899447 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c14584cd-6233-4a85-845a-670843c1d3c8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:10.940774 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:34:10.991075 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-c14584cd-6233-4a85-845a-670843c1d3c8 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:34:10.999062 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-c14584cd-6233-4a85-845a-670843c1d3c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:11.031029 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 882/1767] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:34:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:34:11.033230 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c14584cd-6233-4a85-845a-670843c1d3c8 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:11.055611 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c14584cd-6233-4a85-845a-670843c1d3c8 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:11.076442 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a076352-8033-4cf6-adfb-485b82de3996 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:11.077902 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a076352-8033-4cf6-adfb-485b82de3996 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:11.078652 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a076352-8033-4cf6-adfb-485b82de3996 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:11.079344 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a076352-8033-4cf6-adfb-485b82de3996 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:11.081216 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a076352-8033-4cf6-adfb-485b82de3996 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:11.086640 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a076352-8033-4cf6-adfb-485b82de3996 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:11.110953 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 883/1768] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:34:11 2022] GET /identity/v3/roles => generated 1420 bytes in 74 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:34:11.112585 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a076352-8033-4cf6-adfb-485b82de3996 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:11.133446 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a076352-8033-4cf6-adfb-485b82de3996 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:11.143798 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-651ba9be-1688-4540-a059-8fcb778a17d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:11.144925 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-651ba9be-1688-4540-a059-8fcb778a17d3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:11.145589 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-651ba9be-1688-4540-a059-8fcb778a17d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:11.147769 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-651ba9be-1688-4540-a059-8fcb778a17d3 admin admin] PATH_INFO: `/v3/projects/c033d4a6be5c41bd8eef9e07246330f2/users/29fef06bea154721a1a0ac9a68597432/roles/a6e562cdc96c4be691d638990f8403fd` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:11.227291 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b82c70e0-4abe-476f-8f88-e11f87d0f8dd service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:11.228948 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b82c70e0-4abe-476f-8f88-e11f87d0f8dd service nova] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:11.247085 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-651ba9be-1688-4540-a059-8fcb778a17d3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c033d4a6be5c41bd8eef9e07246330f2, user_id=29fef06bea154721a1a0ac9a68597432, role_id=a6e562cdc96c4be691d638990f8403fd)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:11.249187 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 886/1769] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:34:10 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 966 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:34:11.284744 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-651ba9be-1688-4540-a059-8fcb778a17d3 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:11.379180 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 884/1770] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:34:11 2022] PUT /identity/v3/projects/c033d4a6be5c41bd8eef9e07246330f2/users/29fef06bea154721a1a0ac9a68597432/roles/a6e562cdc96c4be691d638990f8403fd => generated 0 bytes in 268 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:34:11.385073 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b82c70e0-4abe-476f-8f88-e11f87d0f8dd service nova] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:11.411166 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b82c70e0-4abe-476f-8f88-e11f87d0f8dd service nova] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:11.602635 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-651ba9be-1688-4540-a059-8fcb778a17d3 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:11.605400 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-651ba9be-1688-4540-a059-8fcb778a17d3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:11.867389 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95bd8fc7-f6f8-49c4-8b05-7ca8a567a283 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:11.869674 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95bd8fc7-f6f8-49c4-8b05-7ca8a567a283 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:11.870793 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95bd8fc7-f6f8-49c4-8b05-7ca8a567a283 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:11.872651 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95bd8fc7-f6f8-49c4-8b05-7ca8a567a283 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:11.875020 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95bd8fc7-f6f8-49c4-8b05-7ca8a567a283 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:11.879630 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95bd8fc7-f6f8-49c4-8b05-7ca8a567a283 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:11.882541 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-651ba9be-1688-4540-a059-8fcb778a17d3 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:11.896535 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aca16550-4467-4ef6-b311-2768ad923aa0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:11.898402 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aca16550-4467-4ef6-b311-2768ad923aa0 service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:11.901019 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aca16550-4467-4ef6-b311-2768ad923aa0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:11.901724 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aca16550-4467-4ef6-b311-2768ad923aa0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:11.942009 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aca16550-4467-4ef6-b311-2768ad923aa0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:11.944966 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aca16550-4467-4ef6-b311-2768ad923aa0 service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:11.950701 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 887/1771] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:34:11 2022] GET /identity/v3/roles => generated 1420 bytes in 566 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 29 21:34:11.966925 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95bd8fc7-f6f8-49c4-8b05-7ca8a567a283 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:11.976471 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 885/1772] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:34:11 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 375 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 21:34:12.044600 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95bd8fc7-f6f8-49c4-8b05-7ca8a567a283 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:12.067092 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67a93a9c-0fdc-4526-b4cd-c5ba6f9f7638 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:12.068150 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67a93a9c-0fdc-4526-b4cd-c5ba6f9f7638 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:12.068770 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67a93a9c-0fdc-4526-b4cd-c5ba6f9f7638 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:12.069517 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67a93a9c-0fdc-4526-b4cd-c5ba6f9f7638 admin admin] PATH_INFO: `/v3/projects/c033d4a6be5c41bd8eef9e07246330f2/users/29fef06bea154721a1a0ac9a68597432/roles/a6e562cdc96c4be691d638990f8403fd` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:12.079952 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67a93a9c-0fdc-4526-b4cd-c5ba6f9f7638 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c033d4a6be5c41bd8eef9e07246330f2, user_id=29fef06bea154721a1a0ac9a68597432, role_id=a6e562cdc96c4be691d638990f8403fd)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:12.094065 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67a93a9c-0fdc-4526-b4cd-c5ba6f9f7638 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:12.182427 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 888/1773] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:34:11 2022] PUT /identity/v3/projects/c033d4a6be5c41bd8eef9e07246330f2/users/29fef06bea154721a1a0ac9a68597432/roles/a6e562cdc96c4be691d638990f8403fd => generated 0 bytes in 208 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:34:12.194329 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f549ec9-6d97-41d2-a522-92f3f25c834c None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:12.195670 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f549ec9-6d97-41d2-a522-92f3f25c834c None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:12.196192 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f549ec9-6d97-41d2-a522-92f3f25c834c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:12.342061 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-4f549ec9-6d97-41d2-a522-92f3f25c834c None None] MFA Rules not processed for user `29fef06bea154721a1a0ac9a68597432`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:34:12.343356 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-4f549ec9-6d97-41d2-a522-92f3f25c834c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:12.569571 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:34:12.648942 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 886/1774] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:34:12 2022] POST /identity/v3/auth/tokens => generated 3347 bytes in 457 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:34:12.661556 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67a93a9c-0fdc-4526-b4cd-c5ba6f9f7638 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:12.665286 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-67a93a9c-0fdc-4526-b4cd-c5ba6f9f7638 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:12.845248 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67a93a9c-0fdc-4526-b4cd-c5ba6f9f7638 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:12.859245 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2da0606-d8fe-4d7c-80ff-e875dabcb29e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dc226f7ddcb49ecacb51ce57fdfc11b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:12.860082 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2da0606-d8fe-4d7c-80ff-e875dabcb29e service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:12.860388 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2da0606-d8fe-4d7c-80ff-e875dabcb29e service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:12.861513 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2da0606-d8fe-4d7c-80ff-e875dabcb29e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:12.898885 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2da0606-d8fe-4d7c-80ff-e875dabcb29e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:12.924967 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2da0606-d8fe-4d7c-80ff-e875dabcb29e service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:12.971144 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 889/1775] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:34:12 2022] GET /identity/v3/auth/tokens => generated 3347 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:34:13.123188 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afed1138-556b-4c2f-b154-e9b4acef55c2 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:13.123504 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afed1138-556b-4c2f-b154-e9b4acef55c2 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:13.123792 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afed1138-556b-4c2f-b154-e9b4acef55c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:13.218492 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-afed1138-556b-4c2f-b154-e9b4acef55c2 None None] MFA Rules not processed for user `b33a740f1c8a4d9db8d199eff66b76b2`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:34:13.229575 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-afed1138-556b-4c2f-b154-e9b4acef55c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:13.450004 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ccacdfb-0008-4d73-9789-bd89fd1dc623 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:13.450657 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ccacdfb-0008-4d73-9789-bd89fd1dc623 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:13.451073 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ccacdfb-0008-4d73-9789-bd89fd1dc623 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:13.453387 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 890/1776] 10.222.0.90 () {54 vars in 921 bytes} [Mon Aug 29 21:34:13 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 29 21:34:13.575100 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 887/1777] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:34:13 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 453 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:34:13.589477 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ccacdfb-0008-4d73-9789-bd89fd1dc623 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:13.603468 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-2ccacdfb-0008-4d73-9789-bd89fd1dc623 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:13.749566 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afed1138-556b-4c2f-b154-e9b4acef55c2 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:13.902182 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afed1138-556b-4c2f-b154-e9b4acef55c2 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:13.923533 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55dbd0d4-9812-4fba-9c34-7c00a8578e32 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:13.925926 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55dbd0d4-9812-4fba-9c34-7c00a8578e32 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:13.927341 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55dbd0d4-9812-4fba-9c34-7c00a8578e32 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:13.927748 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55dbd0d4-9812-4fba-9c34-7c00a8578e32 admin admin] PATH_INFO: `/v3/users/874d5f9ad9f84d7e8bb644a92e60d110` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:13.933506 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55dbd0d4-9812-4fba-9c34-7c00a8578e32 admin admin] RBAC: Authorizing `identity:delete_user(user_id=874d5f9ad9f84d7e8bb644a92e60d110)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:13.941152 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55dbd0d4-9812-4fba-9c34-7c00a8578e32 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:13.970479 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:34:14.108683 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-55dbd0d4-9812-4fba-9c34-7c00a8578e32 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '874d5f9ad9f84d7e8bb644a92e60d110'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:14.109142 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ccacdfb-0008-4d73-9789-bd89fd1dc623 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:14.128735 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e1e6ea2-d54c-4afd-80ea-8ff40b1607f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:14.129796 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e1e6ea2-d54c-4afd-80ea-8ff40b1607f1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:14.130296 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e1e6ea2-d54c-4afd-80ea-8ff40b1607f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:14.130755 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e1e6ea2-d54c-4afd-80ea-8ff40b1607f1 admin admin] PATH_INFO: `/v3/users/abbce453b48243e28acb1a80df49f513` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:14.132996 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e1e6ea2-d54c-4afd-80ea-8ff40b1607f1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=abbce453b48243e28acb1a80df49f513)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:14.136579 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e1e6ea2-d54c-4afd-80ea-8ff40b1607f1 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:14.153723 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-55dbd0d4-9812-4fba-9c34-7c00a8578e32 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '874d5f9ad9f84d7e8bb644a92e60d110'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:14.181053 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-55dbd0d4-9812-4fba-9c34-7c00a8578e32 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '874d5f9ad9f84d7e8bb644a92e60d110'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:14.186192 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-55dbd0d4-9812-4fba-9c34-7c00a8578e32 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '874d5f9ad9f84d7e8bb644a92e60d110'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:14.188218 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-55dbd0d4-9812-4fba-9c34-7c00a8578e32 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '874d5f9ad9f84d7e8bb644a92e60d110'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:14.270751 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 888/1778] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:34:13 2022] DELETE /identity/v3/users/874d5f9ad9f84d7e8bb644a92e60d110 => generated 0 bytes in 518 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 21:34:14.274916 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f67e2b6-87ae-44ae-a4e8-a13f03dcdc6c None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:14.276523 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f67e2b6-87ae-44ae-a4e8-a13f03dcdc6c None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:14.277731 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f67e2b6-87ae-44ae-a4e8-a13f03dcdc6c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:14.346336 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-6f67e2b6-87ae-44ae-a4e8-a13f03dcdc6c None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:34:14.349042 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-6f67e2b6-87ae-44ae-a4e8-a13f03dcdc6c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:14.351058 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-1e1e6ea2-d54c-4afd-80ea-8ff40b1607f1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'abbce453b48243e28acb1a80df49f513'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:14.377238 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:34:14.390044 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-1e1e6ea2-d54c-4afd-80ea-8ff40b1607f1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'abbce453b48243e28acb1a80df49f513'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:14.391258 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-1e1e6ea2-d54c-4afd-80ea-8ff40b1607f1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'abbce453b48243e28acb1a80df49f513'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:14.394890 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-1e1e6ea2-d54c-4afd-80ea-8ff40b1607f1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'abbce453b48243e28acb1a80df49f513'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:14.416244 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-1e1e6ea2-d54c-4afd-80ea-8ff40b1607f1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'abbce453b48243e28acb1a80df49f513'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:14.439108 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 891/1779] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:34:13 2022] DELETE /identity/v3/users/abbce453b48243e28acb1a80df49f513 => generated 0 bytes in 852 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:34:14.450247 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fff9449-e8d9-4d4e-b872-fcf4cb755a18 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:14.450718 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fff9449-e8d9-4d4e-b872-fcf4cb755a18 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:14.453602 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fff9449-e8d9-4d4e-b872-fcf4cb755a18 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:14.533146 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-9fff9449-e8d9-4d4e-b872-fcf4cb755a18 None None] MFA Rules not processed for user `a35b5b7dea7b421da056f063d2436ffb`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:34:14.535586 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-9fff9449-e8d9-4d4e-b872-fcf4cb755a18 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:14.551852 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 889/1780] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:34:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 280 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:34:14.553707 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ff1bf1d-fe66-4c30-bc63-b85d35dfed27 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:14.554051 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ff1bf1d-fe66-4c30-bc63-b85d35dfed27 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:14.554308 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ff1bf1d-fe66-4c30-bc63-b85d35dfed27 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:14.621768 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-0ff1bf1d-fe66-4c30-bc63-b85d35dfed27 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:34:14.623104 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-0ff1bf1d-fe66-4c30-bc63-b85d35dfed27 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:14.764704 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 890/1781] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:34:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 212 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:34:14.842668 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ff1bf1d-fe66-4c30-bc63-b85d35dfed27 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:14.845039 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-0ff1bf1d-fe66-4c30-bc63-b85d35dfed27 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:14.941191 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 892/1782] 10.222.0.90 () {58 vars in 1018 bytes} [Mon Aug 29 21:34:14 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 501 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:34:14.950939 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fff9449-e8d9-4d4e-b872-fcf4cb755a18 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:14.953154 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-9fff9449-e8d9-4d4e-b872-fcf4cb755a18 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:15.012300 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ff1bf1d-fe66-4c30-bc63-b85d35dfed27 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:15.021289 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ffee977-c3dc-4690-b3a0-a63c60065288 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:15.023131 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ffee977-c3dc-4690-b3a0-a63c60065288 service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:15.023577 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ffee977-c3dc-4690-b3a0-a63c60065288 service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:15.023978 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ffee977-c3dc-4690-b3a0-a63c60065288 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:15.049694 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ffee977-c3dc-4690-b3a0-a63c60065288 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:15.054840 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ffee977-c3dc-4690-b3a0-a63c60065288 service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:15.081605 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 891/1783] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:34:14 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 242 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 21:34:15.236877 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:34:15.258714 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fff9449-e8d9-4d4e-b872-fcf4cb755a18 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:15.291392 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90eec748-9449-431c-aa1d-044989617585 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:15.292415 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90eec748-9449-431c-aa1d-044989617585 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:15.292876 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90eec748-9449-431c-aa1d-044989617585 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:15.293333 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90eec748-9449-431c-aa1d-044989617585 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:15.343128 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90eec748-9449-431c-aa1d-044989617585 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:15.353087 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90eec748-9449-431c-aa1d-044989617585 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:15.379686 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 893/1784] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:34:14 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 438 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:34:16.460753 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ffee977-c3dc-4690-b3a0-a63c60065288 service neutron] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:16.464382 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-7ffee977-c3dc-4690-b3a0-a63c60065288 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:16.520769 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ffee977-c3dc-4690-b3a0-a63c60065288 service neutron] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:16.531924 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4d2fd76-6b5c-41e3-b53e-9b5cc484ef87 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a35b5b7dea7b421da056f063d2436ffb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:16.533047 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d2fd76-6b5c-41e3-b53e-9b5cc484ef87 service nova] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:16.533522 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d2fd76-6b5c-41e3-b53e-9b5cc484ef87 service nova] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:16.534232 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d2fd76-6b5c-41e3-b53e-9b5cc484ef87 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:16.556064 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4d2fd76-6b5c-41e3-b53e-9b5cc484ef87 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:16.559473 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4d2fd76-6b5c-41e3-b53e-9b5cc484ef87 service nova] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:16.596216 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 892/1785] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:34:16 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 137 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:34:18.333666 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90eec748-9449-431c-aa1d-044989617585 service neutron] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:18.337466 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-90eec748-9449-431c-aa1d-044989617585 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:18.397734 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90eec748-9449-431c-aa1d-044989617585 service neutron] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:18.425034 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf11805e-03a4-4e10-9a2c-6a0f0aba3e7b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:18.428702 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf11805e-03a4-4e10-9a2c-6a0f0aba3e7b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:18.429164 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf11805e-03a4-4e10-9a2c-6a0f0aba3e7b admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:18.429657 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf11805e-03a4-4e10-9a2c-6a0f0aba3e7b admin admin] PATH_INFO: `/v3/projects/3287ead9dd7741ccaea942cc2027c7be` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:18.446115 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4d2fd76-6b5c-41e3-b53e-9b5cc484ef87 service nova] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:18.448920 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-e4d2fd76-6b5c-41e3-b53e-9b5cc484ef87 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:18.464222 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf11805e-03a4-4e10-9a2c-6a0f0aba3e7b admin admin] RBAC: Authorizing `identity:delete_project(project_id=3287ead9dd7741ccaea942cc2027c7be)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:18.491482 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf11805e-03a4-4e10-9a2c-6a0f0aba3e7b admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:18.526796 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4d2fd76-6b5c-41e3-b53e-9b5cc484ef87 service nova] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:18.566571 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f9a4a07-f843-48f3-b910-e6bc4544baae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:18.567418 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f9a4a07-f843-48f3-b910-e6bc4544baae admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:18.567705 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f9a4a07-f843-48f3-b910-e6bc4544baae admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:18.568035 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f9a4a07-f843-48f3-b910-e6bc4544baae admin admin] PATH_INFO: `/v3/projects/e02a98407a4b4cc186409f1a8768abfe` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:18.573045 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f9a4a07-f843-48f3-b910-e6bc4544baae admin admin] RBAC: Authorizing `identity:delete_project(project_id=e02a98407a4b4cc186409f1a8768abfe)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:18.585738 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f9a4a07-f843-48f3-b910-e6bc4544baae admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:18.641931 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-bf11805e-03a4-4e10-9a2c-6a0f0aba3e7b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3287ead9dd7741ccaea942cc2027c7be', 'request_id': 'req-bf11805e-03a4-4e10-9a2c-6a0f0aba3e7b'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:18.659553 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-bf11805e-03a4-4e10-9a2c-6a0f0aba3e7b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3287ead9dd7741ccaea942cc2027c7be', 'request_id': 'req-bf11805e-03a4-4e10-9a2c-6a0f0aba3e7b'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:18.668965 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:34:18.688068 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-8f9a4a07-f843-48f3-b910-e6bc4544baae admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e02a98407a4b4cc186409f1a8768abfe', 'request_id': 'req-8f9a4a07-f843-48f3-b910-e6bc4544baae'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:18.693333 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-bf11805e-03a4-4e10-9a2c-6a0f0aba3e7b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3287ead9dd7741ccaea942cc2027c7be', 'request_id': 'req-bf11805e-03a4-4e10-9a2c-6a0f0aba3e7b'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:18.708386 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-8f9a4a07-f843-48f3-b910-e6bc4544baae admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e02a98407a4b4cc186409f1a8768abfe', 'request_id': 'req-8f9a4a07-f843-48f3-b910-e6bc4544baae'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:18.726743 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-bf11805e-03a4-4e10-9a2c-6a0f0aba3e7b admin admin] The token cache is being invalidate because project 3287ead9dd7741ccaea942cc2027c7be was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:34:18.727327 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-bf11805e-03a4-4e10-9a2c-6a0f0aba3e7b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:18.728792 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-bf11805e-03a4-4e10-9a2c-6a0f0aba3e7b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:18.731512 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 894/1786] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:34:18 2022] DELETE /identity/v3/projects/3287ead9dd7741ccaea942cc2027c7be => generated 0 bytes in 399 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:34:18.733280 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c90d5924-14cb-47c7-94c6-15038b7c0759 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:18.733903 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c90d5924-14cb-47c7-94c6-15038b7c0759 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:18.734504 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c90d5924-14cb-47c7-94c6-15038b7c0759 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:18.736744 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 895/1787] 10.222.0.90 () {54 vars in 921 bytes} [Mon Aug 29 21:34:18 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:34:18.737440 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-8f9a4a07-f843-48f3-b910-e6bc4544baae admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e02a98407a4b4cc186409f1a8768abfe', 'request_id': 'req-8f9a4a07-f843-48f3-b910-e6bc4544baae'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:18.738003 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c90d5924-14cb-47c7-94c6-15038b7c0759 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:18.740341 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-c90d5924-14cb-47c7-94c6-15038b7c0759 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:18.767963 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-8f9a4a07-f843-48f3-b910-e6bc4544baae admin admin] The token cache is being invalidate because project e02a98407a4b4cc186409f1a8768abfe was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81434) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:34:18.768480 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-8f9a4a07-f843-48f3-b910-e6bc4544baae admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:18.773998 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-8f9a4a07-f843-48f3-b910-e6bc4544baae admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:18.783015 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 893/1788] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:34:18 2022] DELETE /identity/v3/projects/e02a98407a4b4cc186409f1a8768abfe => generated 0 bytes in 333 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:34:18.786330 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1c8acfb-6d26-4554-8e63-fcc6f127d35b None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:18.786953 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1c8acfb-6d26-4554-8e63-fcc6f127d35b None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:18.788169 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1c8acfb-6d26-4554-8e63-fcc6f127d35b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:18.819965 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-e1c8acfb-6d26-4554-8e63-fcc6f127d35b None None] MFA Rules not processed for user `a35b5b7dea7b421da056f063d2436ffb`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:34:18.823123 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-e1c8acfb-6d26-4554-8e63-fcc6f127d35b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:19.026689 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c90d5924-14cb-47c7-94c6-15038b7c0759 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:19.044394 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f7f8e71-47dc-4579-b860-1b7e6cb28e3e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:19.049973 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f7f8e71-47dc-4579-b860-1b7e6cb28e3e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:19.051070 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f7f8e71-47dc-4579-b860-1b7e6cb28e3e admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:19.052920 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f7f8e71-47dc-4579-b860-1b7e6cb28e3e admin admin] PATH_INFO: `/v3/users/29fef06bea154721a1a0ac9a68597432` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:19.056454 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f7f8e71-47dc-4579-b860-1b7e6cb28e3e admin admin] RBAC: Authorizing `identity:delete_user(user_id=29fef06bea154721a1a0ac9a68597432)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:19.061769 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f7f8e71-47dc-4579-b860-1b7e6cb28e3e admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:19.077853 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 894/1789] 10.222.0.90 () {58 vars in 1018 bytes} [Mon Aug 29 21:34:18 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 292 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:34:19.079452 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1c8acfb-6d26-4554-8e63-fcc6f127d35b None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:19.086622 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-e1c8acfb-6d26-4554-8e63-fcc6f127d35b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:19.140834 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1c8acfb-6d26-4554-8e63-fcc6f127d35b None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:19.175683 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a513f313-aa56-417b-a9dd-36a2c1301046 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:19.180467 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a513f313-aa56-417b-a9dd-36a2c1301046 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:19.182838 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a513f313-aa56-417b-a9dd-36a2c1301046 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:19.183823 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a513f313-aa56-417b-a9dd-36a2c1301046 admin admin] PATH_INFO: `/v3/users/3f6b1a97c07b4090bca1fb483f32a4f6` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:19.192592 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a513f313-aa56-417b-a9dd-36a2c1301046 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3f6b1a97c07b4090bca1fb483f32a4f6)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:19.204365 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a513f313-aa56-417b-a9dd-36a2c1301046 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:19.253511 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-9f7f8e71-47dc-4579-b860-1b7e6cb28e3e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '29fef06bea154721a1a0ac9a68597432'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:19.306684 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-9f7f8e71-47dc-4579-b860-1b7e6cb28e3e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '29fef06bea154721a1a0ac9a68597432'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:19.312498 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-9f7f8e71-47dc-4579-b860-1b7e6cb28e3e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '29fef06bea154721a1a0ac9a68597432'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:19.315509 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-9f7f8e71-47dc-4579-b860-1b7e6cb28e3e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '29fef06bea154721a1a0ac9a68597432'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:19.335278 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-9f7f8e71-47dc-4579-b860-1b7e6cb28e3e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '29fef06bea154721a1a0ac9a68597432'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:19.341268 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=83,keystone:DELETE=16,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:34:19.398561 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 896/1790] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:34:18 2022] DELETE /identity/v3/users/29fef06bea154721a1a0ac9a68597432 => generated 0 bytes in 660 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:34:19.408584 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-a513f313-aa56-417b-a9dd-36a2c1301046 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3f6b1a97c07b4090bca1fb483f32a4f6'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:19.424388 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-a513f313-aa56-417b-a9dd-36a2c1301046 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3f6b1a97c07b4090bca1fb483f32a4f6'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:19.439940 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-a513f313-aa56-417b-a9dd-36a2c1301046 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3f6b1a97c07b4090bca1fb483f32a4f6'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:19.441476 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-a513f313-aa56-417b-a9dd-36a2c1301046 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3f6b1a97c07b4090bca1fb483f32a4f6'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:19.442733 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-a513f313-aa56-417b-a9dd-36a2c1301046 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3f6b1a97c07b4090bca1fb483f32a4f6'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:19.495553 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 895/1791] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:34:19 2022] DELETE /identity/v3/users/3f6b1a97c07b4090bca1fb483f32a4f6 => generated 0 bytes in 413 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:34:19.874640 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f7f8e71-47dc-4579-b860-1b7e6cb28e3e admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:19.877636 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-9f7f8e71-47dc-4579-b860-1b7e6cb28e3e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:20.023721 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f7f8e71-47dc-4579-b860-1b7e6cb28e3e admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:20.043623 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a513f313-aa56-417b-a9dd-36a2c1301046 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:20.046368 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-a513f313-aa56-417b-a9dd-36a2c1301046 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:20.149331 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:34:20.227615 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0c50ed3-8c60-49d5-941d-4c422d0a5d68 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:20.228712 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0c50ed3-8c60-49d5-941d-4c422d0a5d68 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:20.229193 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0c50ed3-8c60-49d5-941d-4c422d0a5d68 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:20.233179 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0c50ed3-8c60-49d5-941d-4c422d0a5d68 admin admin] PATH_INFO: `/v3/projects/ea55bac9f2e7482fb9badfcd17e05741` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:20.239238 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0c50ed3-8c60-49d5-941d-4c422d0a5d68 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ea55bac9f2e7482fb9badfcd17e05741)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:20.245343 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0c50ed3-8c60-49d5-941d-4c422d0a5d68 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:20.282294 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a513f313-aa56-417b-a9dd-36a2c1301046 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:20.302944 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16002aef-b527-4812-b1c4-209806e3a005 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a35b5b7dea7b421da056f063d2436ffb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:20.306754 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16002aef-b527-4812-b1c4-209806e3a005 service nova] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:20.307231 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16002aef-b527-4812-b1c4-209806e3a005 service nova] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:20.307706 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16002aef-b527-4812-b1c4-209806e3a005 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:20.318069 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-16002aef-b527-4812-b1c4-209806e3a005 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:20.386558 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16002aef-b527-4812-b1c4-209806e3a005 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:20.394478 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16002aef-b527-4812-b1c4-209806e3a005 service nova] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:20.421877 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-b0c50ed3-8c60-49d5-941d-4c422d0a5d68 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ea55bac9f2e7482fb9badfcd17e05741', 'request_id': 'req-b0c50ed3-8c60-49d5-941d-4c422d0a5d68'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:20.446834 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-b0c50ed3-8c60-49d5-941d-4c422d0a5d68 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ea55bac9f2e7482fb9badfcd17e05741', 'request_id': 'req-b0c50ed3-8c60-49d5-941d-4c422d0a5d68'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:20.467927 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 896/1792] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:34:20 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 425 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 21:34:20.476613 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-b0c50ed3-8c60-49d5-941d-4c422d0a5d68 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ea55bac9f2e7482fb9badfcd17e05741', 'request_id': 'req-b0c50ed3-8c60-49d5-941d-4c422d0a5d68'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:20.533934 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-b0c50ed3-8c60-49d5-941d-4c422d0a5d68 admin admin] The token cache is being invalidate because project ea55bac9f2e7482fb9badfcd17e05741 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:34:20.534265 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-b0c50ed3-8c60-49d5-941d-4c422d0a5d68 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:20.536078 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-b0c50ed3-8c60-49d5-941d-4c422d0a5d68 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:20.542566 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 897/1793] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:34:19 2022] DELETE /identity/v3/projects/ea55bac9f2e7482fb9badfcd17e05741 => generated 0 bytes in 669 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:34:20.549041 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16002aef-b527-4812-b1c4-209806e3a005 service nova] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:20.551295 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-16002aef-b527-4812-b1c4-209806e3a005 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:20.604594 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16002aef-b527-4812-b1c4-209806e3a005 service nova] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:20.798914 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea5fb54b-b793-465c-bd32-03067c5d63dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:20.799997 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea5fb54b-b793-465c-bd32-03067c5d63dd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:20.800300 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea5fb54b-b793-465c-bd32-03067c5d63dd admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:20.800579 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea5fb54b-b793-465c-bd32-03067c5d63dd admin admin] PATH_INFO: `/v3/users/2dcadec16df6466799d0ffc9da063109` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:20.802179 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea5fb54b-b793-465c-bd32-03067c5d63dd admin admin] RBAC: Authorizing `identity:delete_user(user_id=2dcadec16df6466799d0ffc9da063109)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:20.808675 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea5fb54b-b793-465c-bd32-03067c5d63dd admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:20.932873 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-ea5fb54b-b793-465c-bd32-03067c5d63dd admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2dcadec16df6466799d0ffc9da063109'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:20.950546 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-ea5fb54b-b793-465c-bd32-03067c5d63dd admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2dcadec16df6466799d0ffc9da063109'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:20.968139 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-ea5fb54b-b793-465c-bd32-03067c5d63dd admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2dcadec16df6466799d0ffc9da063109'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:20.969340 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-ea5fb54b-b793-465c-bd32-03067c5d63dd admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2dcadec16df6466799d0ffc9da063109'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:20.970484 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-ea5fb54b-b793-465c-bd32-03067c5d63dd admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2dcadec16df6466799d0ffc9da063109'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:21.016215 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 897/1794] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:34:20 2022] DELETE /identity/v3/users/2dcadec16df6466799d0ffc9da063109 => generated 0 bytes in 469 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:34:21.679269 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0c50ed3-8c60-49d5-941d-4c422d0a5d68 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:21.681578 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-b0c50ed3-8c60-49d5-941d-4c422d0a5d68 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:21.701938 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0c50ed3-8c60-49d5-941d-4c422d0a5d68 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:21.705412 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea5fb54b-b793-465c-bd32-03067c5d63dd admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:21.713925 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-ea5fb54b-b793-465c-bd32-03067c5d63dd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:21.765277 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:INSERT=2,keystone:DELETE=8,keystone:SELECT=90 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:34:21.840215 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b15c1a09-b445-465d-946b-c4da59b3d47b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:21.843701 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b15c1a09-b445-465d-946b-c4da59b3d47b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:21.845193 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b15c1a09-b445-465d-946b-c4da59b3d47b admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:21.847040 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b15c1a09-b445-465d-946b-c4da59b3d47b admin admin] PATH_INFO: `/v3/projects/65263321fac44ae69c91924f4bee07d1` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:21.848674 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b15c1a09-b445-465d-946b-c4da59b3d47b admin admin] RBAC: Authorizing `identity:delete_project(project_id=65263321fac44ae69c91924f4bee07d1)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:21.855256 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b15c1a09-b445-465d-946b-c4da59b3d47b admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:21.925443 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:34:21.968714 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-b15c1a09-b445-465d-946b-c4da59b3d47b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '65263321fac44ae69c91924f4bee07d1', 'request_id': 'req-b15c1a09-b445-465d-946b-c4da59b3d47b'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:21.985222 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-b15c1a09-b445-465d-946b-c4da59b3d47b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '65263321fac44ae69c91924f4bee07d1', 'request_id': 'req-b15c1a09-b445-465d-946b-c4da59b3d47b'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:22.000429 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-b15c1a09-b445-465d-946b-c4da59b3d47b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '65263321fac44ae69c91924f4bee07d1', 'request_id': 'req-b15c1a09-b445-465d-946b-c4da59b3d47b'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:22.004973 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea5fb54b-b793-465c-bd32-03067c5d63dd admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:22.029984 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-b15c1a09-b445-465d-946b-c4da59b3d47b admin admin] The token cache is being invalidate because project 65263321fac44ae69c91924f4bee07d1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:34:22.030652 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-b15c1a09-b445-465d-946b-c4da59b3d47b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:22.033044 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-b15c1a09-b445-465d-946b-c4da59b3d47b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:22.036034 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 898/1795] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:34:21 2022] DELETE /identity/v3/projects/65263321fac44ae69c91924f4bee07d1 => generated 0 bytes in 357 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:34:22.042958 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d5de36a-f7f0-4a68-ae60-04990d50f615 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:22.043514 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d5de36a-f7f0-4a68-ae60-04990d50f615 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:22.043943 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d5de36a-f7f0-4a68-ae60-04990d50f615 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:22.086624 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-0d5de36a-f7f0-4a68-ae60-04990d50f615 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:34:22.088138 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-0d5de36a-f7f0-4a68-ae60-04990d50f615 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:22.109846 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45fbf31d-b003-4ca6-8410-9dba5c5292b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:22.110582 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45fbf31d-b003-4ca6-8410-9dba5c5292b1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:22.110930 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45fbf31d-b003-4ca6-8410-9dba5c5292b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:22.112560 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45fbf31d-b003-4ca6-8410-9dba5c5292b1 admin admin] PATH_INFO: `/v3/projects/c033d4a6be5c41bd8eef9e07246330f2` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:22.114631 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45fbf31d-b003-4ca6-8410-9dba5c5292b1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c033d4a6be5c41bd8eef9e07246330f2)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:22.120252 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45fbf31d-b003-4ca6-8410-9dba5c5292b1 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:22.205002 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-45fbf31d-b003-4ca6-8410-9dba5c5292b1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c033d4a6be5c41bd8eef9e07246330f2', 'request_id': 'req-45fbf31d-b003-4ca6-8410-9dba5c5292b1'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:22.215196 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-45fbf31d-b003-4ca6-8410-9dba5c5292b1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c033d4a6be5c41bd8eef9e07246330f2', 'request_id': 'req-45fbf31d-b003-4ca6-8410-9dba5c5292b1'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:22.225132 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-45fbf31d-b003-4ca6-8410-9dba5c5292b1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c033d4a6be5c41bd8eef9e07246330f2', 'request_id': 'req-45fbf31d-b003-4ca6-8410-9dba5c5292b1'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:22.255385 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 899/1796] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:34:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 213 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:34:22.266281 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9c90c44-a414-4d52-969c-46df90efe77c None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:22.266592 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9c90c44-a414-4d52-969c-46df90efe77c None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:22.267120 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9c90c44-a414-4d52-969c-46df90efe77c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:22.271438 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-45fbf31d-b003-4ca6-8410-9dba5c5292b1 admin admin] The token cache is being invalidate because project c033d4a6be5c41bd8eef9e07246330f2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81434) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:34:22.272714 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-45fbf31d-b003-4ca6-8410-9dba5c5292b1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:22.274197 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-45fbf31d-b003-4ca6-8410-9dba5c5292b1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:22.281440 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 898/1797] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:34:21 2022] DELETE /identity/v3/projects/c033d4a6be5c41bd8eef9e07246330f2 => generated 0 bytes in 573 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:34:22.283350 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93e97654-ae37-4cfd-8c52-688b936ab125 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:22.283751 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93e97654-ae37-4cfd-8c52-688b936ab125 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:22.284124 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93e97654-ae37-4cfd-8c52-688b936ab125 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:22.325992 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-93e97654-ae37-4cfd-8c52-688b936ab125 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:34:22.327346 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-b9c90c44-a414-4d52-969c-46df90efe77c None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:34:22.328707 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-93e97654-ae37-4cfd-8c52-688b936ab125 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:22.330345 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-b9c90c44-a414-4d52-969c-46df90efe77c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:22.524374 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 899/1798] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:34:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 242 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:34:22.525620 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:34:22.533907 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85c25fb7-4cd3-4880-a987-967084a3fcf0 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:22.534414 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85c25fb7-4cd3-4880-a987-967084a3fcf0 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:22.534885 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85c25fb7-4cd3-4880-a987-967084a3fcf0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:22.548340 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 900/1799] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:34:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 285 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 21:34:22.549855 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9c90c44-a414-4d52-969c-46df90efe77c None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:22.609477 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9c90c44-a414-4d52-969c-46df90efe77c None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:22.623255 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c06b513e-e506-4fdb-be26-2ce65abe9ef5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:22.623925 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c06b513e-e506-4fdb-be26-2ce65abe9ef5 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:22.633319 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c06b513e-e506-4fdb-be26-2ce65abe9ef5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:22.633319 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c06b513e-e506-4fdb-be26-2ce65abe9ef5 admin admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:22.633637 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-85c25fb7-4cd3-4880-a987-967084a3fcf0 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:34:22.633759 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c06b513e-e506-4fdb-be26-2ce65abe9ef5 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:34:22.633759 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c06b513e-e506-4fdb-be26-2ce65abe9ef5 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:22.634030 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-85c25fb7-4cd3-4880-a987-967084a3fcf0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:22.634331 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c06b513e-e506-4fdb-be26-2ce65abe9ef5 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:22.666887 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 901/1800] 10.222.0.90 () {58 vars in 1153 bytes} [Mon Aug 29 21:34:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 117 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:34:22.675298 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c06b513e-e506-4fdb-be26-2ce65abe9ef5 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:22.698434 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 900/1801] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:34:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:34:22.698434 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85c25fb7-4cd3-4880-a987-967084a3fcf0 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:22.728776 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c06b513e-e506-4fdb-be26-2ce65abe9ef5 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:22.742195 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85c25fb7-4cd3-4880-a987-967084a3fcf0 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:22.746479 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cee2dec3-bd94-464f-8049-bf11b69b9e45 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:22.747377 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cee2dec3-bd94-464f-8049-bf11b69b9e45 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:22.747735 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cee2dec3-bd94-464f-8049-bf11b69b9e45 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:22.748087 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cee2dec3-bd94-464f-8049-bf11b69b9e45 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:22.748852 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cee2dec3-bd94-464f-8049-bf11b69b9e45 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:22.752175 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cee2dec3-bd94-464f-8049-bf11b69b9e45 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:22.761670 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ee88a17-233c-4f59-8d68-a3efd085c040 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:22.762412 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ee88a17-233c-4f59-8d68-a3efd085c040 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:22.762690 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ee88a17-233c-4f59-8d68-a3efd085c040 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:22.763067 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ee88a17-233c-4f59-8d68-a3efd085c040 admin admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:22.763742 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ee88a17-233c-4f59-8d68-a3efd085c040 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:34:22.764053 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ee88a17-233c-4f59-8d68-a3efd085c040 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:22.765690 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ee88a17-233c-4f59-8d68-a3efd085c040 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:22.776460 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:34:22.803793 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 901/1802] 10.222.0.90 () {58 vars in 1153 bytes} [Mon Aug 29 21:34:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 111 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:34:22.805812 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ee88a17-233c-4f59-8d68-a3efd085c040 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:22.840085 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 902/1803] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:34:22 2022] POST /identity/v3/projects => generated 377 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:34:22.840785 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cee2dec3-bd94-464f-8049-bf11b69b9e45 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:22.851874 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-cee2dec3-bd94-464f-8049-bf11b69b9e45 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:22.857762 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ee88a17-233c-4f59-8d68-a3efd085c040 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:22.994972 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82d031c5-5358-43f9-8cb7-dcceb8cbacec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:22.996495 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82d031c5-5358-43f9-8cb7-dcceb8cbacec admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:22.996790 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82d031c5-5358-43f9-8cb7-dcceb8cbacec admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:22.997368 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82d031c5-5358-43f9-8cb7-dcceb8cbacec admin admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:22.998675 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82d031c5-5358-43f9-8cb7-dcceb8cbacec admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:23.000675 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82d031c5-5358-43f9-8cb7-dcceb8cbacec admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:23.002604 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cee2dec3-bd94-464f-8049-bf11b69b9e45 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:23.015419 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bc33262-e511-446f-92d9-ec3c9cdf8336 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:23.016668 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bc33262-e511-446f-92d9-ec3c9cdf8336 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:23.017008 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bc33262-e511-446f-92d9-ec3c9cdf8336 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:23.017348 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bc33262-e511-446f-92d9-ec3c9cdf8336 admin admin] PATH_INFO: `/v3/projects/c64b966440f5415bb9d75cb47ac1ecae` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:23.018834 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bc33262-e511-446f-92d9-ec3c9cdf8336 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c64b966440f5415bb9d75cb47ac1ecae)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:23.021212 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bc33262-e511-446f-92d9-ec3c9cdf8336 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:23.081840 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 902/1804] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:34:22 2022] POST /identity/v3/projects => generated 397 bytes in 277 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:34:23.083412 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82d031c5-5358-43f9-8cb7-dcceb8cbacec admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:23.125562 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-9bc33262-e511-446f-92d9-ec3c9cdf8336 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c64b966440f5415bb9d75cb47ac1ecae', 'request_id': 'req-9bc33262-e511-446f-92d9-ec3c9cdf8336'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:23.136058 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-9bc33262-e511-446f-92d9-ec3c9cdf8336 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c64b966440f5415bb9d75cb47ac1ecae', 'request_id': 'req-9bc33262-e511-446f-92d9-ec3c9cdf8336'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:23.150723 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-9bc33262-e511-446f-92d9-ec3c9cdf8336 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c64b966440f5415bb9d75cb47ac1ecae', 'request_id': 'req-9bc33262-e511-446f-92d9-ec3c9cdf8336'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:23.179225 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-9bc33262-e511-446f-92d9-ec3c9cdf8336 admin admin] The token cache is being invalidate because project c64b966440f5415bb9d75cb47ac1ecae was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:34:23.179746 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-9bc33262-e511-446f-92d9-ec3c9cdf8336 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:23.182809 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-9bc33262-e511-446f-92d9-ec3c9cdf8336 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:23.186167 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 903/1805] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:34:22 2022] DELETE /identity/v3/projects/c64b966440f5415bb9d75cb47ac1ecae => generated 0 bytes in 346 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:34:23.195236 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bc33262-e511-446f-92d9-ec3c9cdf8336 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:23.198436 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-9bc33262-e511-446f-92d9-ec3c9cdf8336 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:23.243385 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82d031c5-5358-43f9-8cb7-dcceb8cbacec admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:23.369442 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5a3c225-b92d-409c-bb3f-614c4cc5521f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:23.370114 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5a3c225-b92d-409c-bb3f-614c4cc5521f admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:23.370399 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5a3c225-b92d-409c-bb3f-614c4cc5521f admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:23.370684 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5a3c225-b92d-409c-bb3f-614c4cc5521f admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:23.371339 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5a3c225-b92d-409c-bb3f-614c4cc5521f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:23.373287 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5a3c225-b92d-409c-bb3f-614c4cc5521f admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:23.437599 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 903/1806] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:34:23 2022] POST /identity/v3/users => generated 289 bytes in 355 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:34:23.445418 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-855e213f-a0ab-4c88-abd4-e5dc82ec9d45 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:23.445909 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-855e213f-a0ab-4c88-abd4-e5dc82ec9d45 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:23.446229 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:34:23.453319 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-855e213f-a0ab-4c88-abd4-e5dc82ec9d45 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:23.458462 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bc33262-e511-446f-92d9-ec3c9cdf8336 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:23.475924 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-366b064d-55b8-4232-b8a0-286f97582cef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:23.478154 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-366b064d-55b8-4232-b8a0-286f97582cef admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:23.478642 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-366b064d-55b8-4232-b8a0-286f97582cef admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:23.479132 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-366b064d-55b8-4232-b8a0-286f97582cef admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:23.482578 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-366b064d-55b8-4232-b8a0-286f97582cef admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:23.493544 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-366b064d-55b8-4232-b8a0-286f97582cef admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:23.533015 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-855e213f-a0ab-4c88-abd4-e5dc82ec9d45 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:34:23.543582 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-855e213f-a0ab-4c88-abd4-e5dc82ec9d45 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:23.637832 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 904/1807] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:34:23 2022] POST /identity/v3/users => generated 299 bytes in 447 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:34:23.637832 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-366b064d-55b8-4232-b8a0-286f97582cef admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:23.638225 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 904/1808] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:34:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 193 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:34:23.638897 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-366b064d-55b8-4232-b8a0-286f97582cef admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:23.640183 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-855e213f-a0ab-4c88-abd4-e5dc82ec9d45 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:23.679414 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-855e213f-a0ab-4c88-abd4-e5dc82ec9d45 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:23.691491 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-366b064d-55b8-4232-b8a0-286f97582cef admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:23.702685 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb52f19d-ac2f-47f0-913d-1ff071115348 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:23.703245 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb52f19d-ac2f-47f0-913d-1ff071115348 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:23.703741 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb52f19d-ac2f-47f0-913d-1ff071115348 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:23.704279 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb52f19d-ac2f-47f0-913d-1ff071115348 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:23.706922 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb52f19d-ac2f-47f0-913d-1ff071115348 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:23.715113 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb52f19d-ac2f-47f0-913d-1ff071115348 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:23.722672 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:34:23.736889 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c95bf89-ae1c-4683-ac8b-fd27c2c844e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:23.744588 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c95bf89-ae1c-4683-ac8b-fd27c2c844e6 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:23.745169 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 905/1809] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:34:23 2022] GET /identity/v3/roles => generated 1420 bytes in 106 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 29 21:34:23.747686 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5860dd1b-9837-4b29-ac54-c0791b434fb2 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:23.747976 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5860dd1b-9837-4b29-ac54-c0791b434fb2 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:23.748248 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5860dd1b-9837-4b29-ac54-c0791b434fb2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:23.749279 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c95bf89-ae1c-4683-ac8b-fd27c2c844e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:23.749554 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c95bf89-ae1c-4683-ac8b-fd27c2c844e6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:23.750289 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c95bf89-ae1c-4683-ac8b-fd27c2c844e6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:23.752028 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c95bf89-ae1c-4683-ac8b-fd27c2c844e6 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:23.796161 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 905/1810] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:34:23 2022] GET /identity/v3/roles => generated 1420 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:34:23.800155 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c95bf89-ae1c-4683-ac8b-fd27c2c844e6 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:23.820226 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-5860dd1b-9837-4b29-ac54-c0791b434fb2 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:34:23.825759 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-5860dd1b-9837-4b29-ac54-c0791b434fb2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:23.826590 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c95bf89-ae1c-4683-ac8b-fd27c2c844e6 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:23.837157 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 906/1811] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:34:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:34:23.837157 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5860dd1b-9837-4b29-ac54-c0791b434fb2 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:23.844289 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e880559f-d487-43ac-9db2-93f461433b71 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:23.845209 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e880559f-d487-43ac-9db2-93f461433b71 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:23.849490 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5860dd1b-9837-4b29-ac54-c0791b434fb2 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:23.850245 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e880559f-d487-43ac-9db2-93f461433b71 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:23.850786 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e880559f-d487-43ac-9db2-93f461433b71 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:23.852234 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e880559f-d487-43ac-9db2-93f461433b71 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:23.859208 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e880559f-d487-43ac-9db2-93f461433b71 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:23.867796 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a2778c8-32d7-4ef9-a882-f8b67bd939e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:23.869875 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a2778c8-32d7-4ef9-a882-f8b67bd939e7 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:23.870913 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a2778c8-32d7-4ef9-a882-f8b67bd939e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:23.872373 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a2778c8-32d7-4ef9-a882-f8b67bd939e7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:23.873414 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a2778c8-32d7-4ef9-a882-f8b67bd939e7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:23.886827 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 906/1812] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:34:23 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:34:23.886827 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-068f4602-f77b-485b-ae11-ba8a5c48c408 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:23.886827 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-068f4602-f77b-485b-ae11-ba8a5c48c408 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:23.886827 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-068f4602-f77b-485b-ae11-ba8a5c48c408 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:23.888305 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a2778c8-32d7-4ef9-a882-f8b67bd939e7 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:23.902133 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 907/1813] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:34:23 2022] GET /identity/v3/roles => generated 1420 bytes in 69 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:34:23.903153 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a2778c8-32d7-4ef9-a882-f8b67bd939e7 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:23.907371 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-068f4602-f77b-485b-ae11-ba8a5c48c408 None None] MFA Rules not processed for user `bbbfb39899b04668ac1b7ff28d18e839`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:34:23.908665 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-068f4602-f77b-485b-ae11-ba8a5c48c408 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:23.916278 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a2778c8-32d7-4ef9-a882-f8b67bd939e7 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:23.923382 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16cbfa40-0180-4de0-979f-6e91bac22708 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:23.923976 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16cbfa40-0180-4de0-979f-6e91bac22708 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:23.924268 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16cbfa40-0180-4de0-979f-6e91bac22708 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:23.925276 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16cbfa40-0180-4de0-979f-6e91bac22708 admin admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:23.925276 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16cbfa40-0180-4de0-979f-6e91bac22708 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:34:23.925521 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16cbfa40-0180-4de0-979f-6e91bac22708 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:23.927238 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16cbfa40-0180-4de0-979f-6e91bac22708 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:23.950824 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 908/1814] 10.222.0.90 () {58 vars in 1153 bytes} [Mon Aug 29 21:34:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:34:23.952169 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16cbfa40-0180-4de0-979f-6e91bac22708 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:23.977382 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16cbfa40-0180-4de0-979f-6e91bac22708 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:23.982648 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4295e6a-cec6-4cc8-8e98-7351f338729a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:23.983431 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4295e6a-cec6-4cc8-8e98-7351f338729a admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:23.983604 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4295e6a-cec6-4cc8-8e98-7351f338729a admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:23.983883 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4295e6a-cec6-4cc8-8e98-7351f338729a admin admin] PATH_INFO: `/v3/projects/2377a7c29f31404cb421097da62aae7b/users/8a9f06d9417f4e919aa63807fb7890aa/roles/02e242d5520740d4b77a5f4008c7eb57` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:24.002568 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4295e6a-cec6-4cc8-8e98-7351f338729a admin admin] RBAC: Authorizing `identity:create_grant(project_id=2377a7c29f31404cb421097da62aae7b, user_id=8a9f06d9417f4e919aa63807fb7890aa, role_id=02e242d5520740d4b77a5f4008c7eb57)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:24.005213 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4295e6a-cec6-4cc8-8e98-7351f338729a admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:24.009640 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 907/1815] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:34:23 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:34:24.010588 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-068f4602-f77b-485b-ae11-ba8a5c48c408 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:24.024854 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-068f4602-f77b-485b-ae11-ba8a5c48c408 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:24.031185 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 909/1816] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:34:23 2022] PUT /identity/v3/projects/2377a7c29f31404cb421097da62aae7b/users/8a9f06d9417f4e919aa63807fb7890aa/roles/02e242d5520740d4b77a5f4008c7eb57 => generated 0 bytes in 80 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:34:24.031926 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4295e6a-cec6-4cc8-8e98-7351f338729a admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:24.147916 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4295e6a-cec6-4cc8-8e98-7351f338729a admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:24.156115 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-577cf9d5-1b24-46d7-ac7b-6a2c30cba299 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:24.158229 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-577cf9d5-1b24-46d7-ac7b-6a2c30cba299 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:24.158743 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-577cf9d5-1b24-46d7-ac7b-6a2c30cba299 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:24.159212 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-577cf9d5-1b24-46d7-ac7b-6a2c30cba299 admin admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:24.160391 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-577cf9d5-1b24-46d7-ac7b-6a2c30cba299 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:24.165297 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-577cf9d5-1b24-46d7-ac7b-6a2c30cba299 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:24.166526 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d80f7b1-89f6-49b1-84ef-a8415631fb55 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:24.168070 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d80f7b1-89f6-49b1-84ef-a8415631fb55 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:24.170422 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d80f7b1-89f6-49b1-84ef-a8415631fb55 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:24.170966 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d80f7b1-89f6-49b1-84ef-a8415631fb55 admin admin] PATH_INFO: `/v3/projects/78a47c9c29634eeea2f1675ee555bb04/users/518273de70ab4b5c9bda2fcb541610bd/roles/02e242d5520740d4b77a5f4008c7eb57` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:24.215411 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d80f7b1-89f6-49b1-84ef-a8415631fb55 admin admin] RBAC: Authorizing `identity:create_grant(project_id=78a47c9c29634eeea2f1675ee555bb04, user_id=518273de70ab4b5c9bda2fcb541610bd, role_id=02e242d5520740d4b77a5f4008c7eb57)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:24.221282 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d80f7b1-89f6-49b1-84ef-a8415631fb55 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:24.230134 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 910/1817] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:34:24 2022] POST /identity/v3/projects => generated 365 bytes in 198 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:34:24.233218 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-484f33a4-87f4-43a8-b27d-872f39c82150 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:24.233818 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-484f33a4-87f4-43a8-b27d-872f39c82150 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:24.234466 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-484f33a4-87f4-43a8-b27d-872f39c82150 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:24.258402 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 908/1818] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:34:24 2022] PUT /identity/v3/projects/78a47c9c29634eeea2f1675ee555bb04/users/518273de70ab4b5c9bda2fcb541610bd/roles/02e242d5520740d4b77a5f4008c7eb57 => generated 0 bytes in 248 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:34:24.259896 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d80f7b1-89f6-49b1-84ef-a8415631fb55 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:24.290096 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-484f33a4-87f4-43a8-b27d-872f39c82150 None None] MFA Rules not processed for user `8a9f06d9417f4e919aa63807fb7890aa`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:34:24.291325 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-484f33a4-87f4-43a8-b27d-872f39c82150 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:24.335683 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:34:24.360262 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d80f7b1-89f6-49b1-84ef-a8415631fb55 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:24.370118 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4be4caff-ea7d-4e15-a9e1-78a93203bbd8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:24.370859 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4be4caff-ea7d-4e15-a9e1-78a93203bbd8 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:24.371575 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4be4caff-ea7d-4e15-a9e1-78a93203bbd8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:24.371943 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4be4caff-ea7d-4e15-a9e1-78a93203bbd8 admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:24.372983 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4be4caff-ea7d-4e15-a9e1-78a93203bbd8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:24.376547 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4be4caff-ea7d-4e15-a9e1-78a93203bbd8 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:24.420203 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 911/1819] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:34:24 2022] POST /identity/v3/auth/tokens => generated 3274 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:34:24.423057 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc364e1-6d46-4ca7-a7fb-d24e40858255 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:24.423806 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc364e1-6d46-4ca7-a7fb-d24e40858255 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:24.424727 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc364e1-6d46-4ca7-a7fb-d24e40858255 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:24.426408 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 909/1820] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:34:24 2022] POST /identity/v3/users => generated 283 bytes in 167 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:34:24.427403 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4be4caff-ea7d-4e15-a9e1-78a93203bbd8 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:24.447194 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4be4caff-ea7d-4e15-a9e1-78a93203bbd8 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:24.453759 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c1dace3-8df8-4deb-aa9f-3c6ba73e34ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:24.454469 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c1dace3-8df8-4deb-aa9f-3c6ba73e34ca admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:24.455127 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c1dace3-8df8-4deb-aa9f-3c6ba73e34ca admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:24.455415 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c1dace3-8df8-4deb-aa9f-3c6ba73e34ca admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:24.456084 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c1dace3-8df8-4deb-aa9f-3c6ba73e34ca admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:24.458328 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c1dace3-8df8-4deb-aa9f-3c6ba73e34ca admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:24.494803 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-9fc364e1-6d46-4ca7-a7fb-d24e40858255 None None] MFA Rules not processed for user `518273de70ab4b5c9bda2fcb541610bd`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:34:24.499088 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-9fc364e1-6d46-4ca7-a7fb-d24e40858255 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:24.515107 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 910/1821] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:34:24 2022] POST /identity/v3/projects => generated 401 bytes in 88 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:34:24.515815 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:34:24.516632 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c1dace3-8df8-4deb-aa9f-3c6ba73e34ca admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:24.713602 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 912/1822] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:34:24 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 292 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:34:24.714825 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fc364e1-6d46-4ca7-a7fb-d24e40858255 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:24.733463 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c1dace3-8df8-4deb-aa9f-3c6ba73e34ca admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:24.746625 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fc364e1-6d46-4ca7-a7fb-d24e40858255 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:24.747184 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6570c6a5-662d-43de-a76d-31e0a5f66615 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:24.747550 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6570c6a5-662d-43de-a76d-31e0a5f66615 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:24.748370 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6570c6a5-662d-43de-a76d-31e0a5f66615 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:24.748765 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6570c6a5-662d-43de-a76d-31e0a5f66615 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:24.751643 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6570c6a5-662d-43de-a76d-31e0a5f66615 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:24.755414 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6570c6a5-662d-43de-a76d-31e0a5f66615 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:24.757330 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62df1dca-07d2-457f-9103-19650559e62e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:24.758007 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62df1dca-07d2-457f-9103-19650559e62e admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:24.758295 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62df1dca-07d2-457f-9103-19650559e62e admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:24.758569 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62df1dca-07d2-457f-9103-19650559e62e admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:24.759231 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62df1dca-07d2-457f-9103-19650559e62e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:24.761378 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62df1dca-07d2-457f-9103-19650559e62e admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:24.794968 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 911/1823] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:34:24 2022] GET /identity/v3/roles => generated 1420 bytes in 279 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:34:24.796730 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6570c6a5-662d-43de-a76d-31e0a5f66615 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:24.814219 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6570c6a5-662d-43de-a76d-31e0a5f66615 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:24.814821 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 913/1824] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:34:24 2022] POST /identity/v3/users => generated 307 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:34:24.823568 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62df1dca-07d2-457f-9103-19650559e62e admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:24.826458 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5221670e-003b-4a70-abed-5c4a906a0d50 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:24.827109 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5221670e-003b-4a70-abed-5c4a906a0d50 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:24.827388 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5221670e-003b-4a70-abed-5c4a906a0d50 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:24.827664 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5221670e-003b-4a70-abed-5c4a906a0d50 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:24.828404 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5221670e-003b-4a70-abed-5c4a906a0d50 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:24.830696 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5221670e-003b-4a70-abed-5c4a906a0d50 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:24.845002 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62df1dca-07d2-457f-9103-19650559e62e admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:24.871193 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adf16f10-9f02-4f6a-9f4e-a2d490dd3701 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:24.873551 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adf16f10-9f02-4f6a-9f4e-a2d490dd3701 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:24.888293 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adf16f10-9f02-4f6a-9f4e-a2d490dd3701 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:24.892859 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adf16f10-9f02-4f6a-9f4e-a2d490dd3701 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:24.893256 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adf16f10-9f02-4f6a-9f4e-a2d490dd3701 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:24.895371 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adf16f10-9f02-4f6a-9f4e-a2d490dd3701 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:24.935835 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 914/1825] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:34:24 2022] GET /identity/v3/roles => generated 1420 bytes in 112 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:34:24.937681 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adf16f10-9f02-4f6a-9f4e-a2d490dd3701 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:24.938692 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 912/1826] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:34:24 2022] POST /identity/v3/projects => generated 379 bytes in 143 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:34:24.940728 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35af16a3-bf91-4d5d-9255-fe21d7bf7675 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:24.941015 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35af16a3-bf91-4d5d-9255-fe21d7bf7675 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:24.941292 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35af16a3-bf91-4d5d-9255-fe21d7bf7675 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:24.948221 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 913/1827] 10.222.0.90 () {54 vars in 921 bytes} [Mon Aug 29 21:34:24 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:34:24.950244 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35af16a3-bf91-4d5d-9255-fe21d7bf7675 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:24.967514 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adf16f10-9f02-4f6a-9f4e-a2d490dd3701 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:25.113161 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7179ea0-1ba1-4bd3-8f52-2ad8c55c8f3f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:25.114249 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7179ea0-1ba1-4bd3-8f52-2ad8c55c8f3f admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:25.117612 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7179ea0-1ba1-4bd3-8f52-2ad8c55c8f3f admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:25.118029 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7179ea0-1ba1-4bd3-8f52-2ad8c55c8f3f admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:25.118890 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7179ea0-1ba1-4bd3-8f52-2ad8c55c8f3f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:25.124683 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7179ea0-1ba1-4bd3-8f52-2ad8c55c8f3f admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:25.142711 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35af16a3-bf91-4d5d-9255-fe21d7bf7675 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:25.165648 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eedc958-bc16-4bf5-9a4c-de63e271f75f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:25.167279 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eedc958-bc16-4bf5-9a4c-de63e271f75f admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:25.170038 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eedc958-bc16-4bf5-9a4c-de63e271f75f admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:25.170276 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 915/1828] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:34:24 2022] GET /identity/v3/roles => generated 1420 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:34:25.170851 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eedc958-bc16-4bf5-9a4c-de63e271f75f admin admin] PATH_INFO: `/v3/projects/fa8d6535c8bb407cacdcd3b1acc172cb/users/9110bb38ab0b46818feadb1ab031892c/roles/02e242d5520740d4b77a5f4008c7eb57` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:25.171577 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7179ea0-1ba1-4bd3-8f52-2ad8c55c8f3f admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:25.214873 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7179ea0-1ba1-4bd3-8f52-2ad8c55c8f3f admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:25.233871 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e0590f2-7a92-43ec-90c2-e04fd68f8c3b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:25.235752 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e0590f2-7a92-43ec-90c2-e04fd68f8c3b admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:25.235752 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e0590f2-7a92-43ec-90c2-e04fd68f8c3b admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:25.238200 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e0590f2-7a92-43ec-90c2-e04fd68f8c3b admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:25.242459 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e0590f2-7a92-43ec-90c2-e04fd68f8c3b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:25.244580 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e0590f2-7a92-43ec-90c2-e04fd68f8c3b admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:25.245400 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eedc958-bc16-4bf5-9a4c-de63e271f75f admin admin] RBAC: Authorizing `identity:create_grant(project_id=fa8d6535c8bb407cacdcd3b1acc172cb, user_id=9110bb38ab0b46818feadb1ab031892c, role_id=02e242d5520740d4b77a5f4008c7eb57)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:25.248610 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eedc958-bc16-4bf5-9a4c-de63e271f75f admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:25.293629 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 914/1829] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:34:24 2022] PUT /identity/v3/projects/fa8d6535c8bb407cacdcd3b1acc172cb/users/9110bb38ab0b46818feadb1ab031892c/roles/02e242d5520740d4b77a5f4008c7eb57 => generated 0 bytes in 345 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:34:25.305612 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89ddc3d4-07cd-4efa-ad11-084471355aa5 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:25.305612 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89ddc3d4-07cd-4efa-ad11-084471355aa5 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:25.305612 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89ddc3d4-07cd-4efa-ad11-084471355aa5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:25.323483 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 916/1830] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:34:25 2022] POST /identity/v3/users => generated 290 bytes in 151 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:34:25.325068 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e0590f2-7a92-43ec-90c2-e04fd68f8c3b admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:25.374879 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e0590f2-7a92-43ec-90c2-e04fd68f8c3b admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:25.378431 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-89ddc3d4-07cd-4efa-ad11-084471355aa5 None None] MFA Rules not processed for user `a35b5b7dea7b421da056f063d2436ffb`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:34:25.379874 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-89ddc3d4-07cd-4efa-ad11-084471355aa5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:25.403946 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:34:25.532462 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ca25b08-d403-4b0f-a71a-b44178c657a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:25.533480 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ca25b08-d403-4b0f-a71a-b44178c657a7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:25.533985 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ca25b08-d403-4b0f-a71a-b44178c657a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:25.534477 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ca25b08-d403-4b0f-a71a-b44178c657a7 admin admin] PATH_INFO: `/v3/projects/2300c43f86074c3b9babc975addbc686/users/4dcfb6941df346f092934f928e5380c8/roles/a6e562cdc96c4be691d638990f8403fd` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:25.540710 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:34:25.565422 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 915/1831] 10.222.0.90 () {58 vars in 1018 bytes} [Mon Aug 29 21:34:25 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 271 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:34:25.566921 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71aeeced-43c3-4579-bc82-4e90155c7225 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:25.567261 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71aeeced-43c3-4579-bc82-4e90155c7225 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:25.567667 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71aeeced-43c3-4579-bc82-4e90155c7225 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:25.576786 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ca25b08-d403-4b0f-a71a-b44178c657a7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2300c43f86074c3b9babc975addbc686, user_id=4dcfb6941df346f092934f928e5380c8, role_id=a6e562cdc96c4be691d638990f8403fd)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:25.581942 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ca25b08-d403-4b0f-a71a-b44178c657a7 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:25.612050 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 917/1832] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:34:25 2022] PUT /identity/v3/projects/2300c43f86074c3b9babc975addbc686/users/4dcfb6941df346f092934f928e5380c8/roles/a6e562cdc96c4be691d638990f8403fd => generated 0 bytes in 288 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:34:25.619955 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ca25b08-d403-4b0f-a71a-b44178c657a7 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:25.624234 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-71aeeced-43c3-4579-bc82-4e90155c7225 None None] MFA Rules not processed for user `9110bb38ab0b46818feadb1ab031892c`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:34:25.626370 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-71aeeced-43c3-4579-bc82-4e90155c7225 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:25.778336 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ca25b08-d403-4b0f-a71a-b44178c657a7 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:25.785496 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9b2939b-7209-4abd-8c3f-f86463d6ea97 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:25.786695 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b2939b-7209-4abd-8c3f-f86463d6ea97 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:25.787623 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b2939b-7209-4abd-8c3f-f86463d6ea97 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:25.788811 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b2939b-7209-4abd-8c3f-f86463d6ea97 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:25.789909 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9b2939b-7209-4abd-8c3f-f86463d6ea97 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:25.792231 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9b2939b-7209-4abd-8c3f-f86463d6ea97 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:25.806892 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 918/1833] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:34:25 2022] GET /identity/v3/roles => generated 1420 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:34:25.808086 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9b2939b-7209-4abd-8c3f-f86463d6ea97 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:25.824450 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9b2939b-7209-4abd-8c3f-f86463d6ea97 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:25.831189 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 916/1834] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:34:25 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 265 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:34:25.832786 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71aeeced-43c3-4579-bc82-4e90155c7225 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:25.838836 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-71aeeced-43c3-4579-bc82-4e90155c7225 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:25.847380 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43daf139-4a49-46e4-a665-3991a299353f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:25.848038 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43daf139-4a49-46e4-a665-3991a299353f admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:25.848331 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43daf139-4a49-46e4-a665-3991a299353f admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:25.848615 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43daf139-4a49-46e4-a665-3991a299353f admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:25.849308 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43daf139-4a49-46e4-a665-3991a299353f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:25.851624 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43daf139-4a49-46e4-a665-3991a299353f admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:25.883525 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 919/1835] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:34:25 2022] GET /identity/v3/roles => generated 1420 bytes in 76 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:34:25.887825 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43daf139-4a49-46e4-a665-3991a299353f admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:25.922730 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43daf139-4a49-46e4-a665-3991a299353f admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:25.938051 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bc0295f-13b6-4664-9cc2-0ccd457344c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:25.938689 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bc0295f-13b6-4664-9cc2-0ccd457344c4 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:25.938973 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bc0295f-13b6-4664-9cc2-0ccd457344c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:25.939324 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bc0295f-13b6-4664-9cc2-0ccd457344c4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:25.940049 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bc0295f-13b6-4664-9cc2-0ccd457344c4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:25.946462 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bc0295f-13b6-4664-9cc2-0ccd457344c4 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:25.989563 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 920/1836] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:34:25 2022] GET /identity/v3/roles => generated 1420 bytes in 105 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:34:25.997656 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bc0295f-13b6-4664-9cc2-0ccd457344c4 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:26.023271 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bc0295f-13b6-4664-9cc2-0ccd457344c4 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:26.028903 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71aeeced-43c3-4579-bc82-4e90155c7225 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:26.041978 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-059d926a-9368-4c86-872e-a9d81b44bc02 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a35b5b7dea7b421da056f063d2436ffb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:26.043322 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-059d926a-9368-4c86-872e-a9d81b44bc02 service nova] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:26.043864 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-059d926a-9368-4c86-872e-a9d81b44bc02 service nova] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:26.044576 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-059d926a-9368-4c86-872e-a9d81b44bc02 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:26.062677 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-059d926a-9368-4c86-872e-a9d81b44bc02 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:26.063351 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ca623a4-2077-4a03-a172-acf3d852ee92 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:26.064382 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ca623a4-2077-4a03-a172-acf3d852ee92 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:26.064865 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ca623a4-2077-4a03-a172-acf3d852ee92 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:26.065341 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ca623a4-2077-4a03-a172-acf3d852ee92 admin admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:26.068757 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-059d926a-9368-4c86-872e-a9d81b44bc02 service nova] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:26.076385 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ca623a4-2077-4a03-a172-acf3d852ee92 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:26.080330 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ca623a4-2077-4a03-a172-acf3d852ee92 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:26.087007 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 917/1837] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:34:25 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 255 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:34:26.088448 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-059d926a-9368-4c86-872e-a9d81b44bc02 service nova] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:26.104885 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-059d926a-9368-4c86-872e-a9d81b44bc02 service nova] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:26.124774 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30665a0f-4d2d-49db-bae5-c413563cd151 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:26.127323 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30665a0f-4d2d-49db-bae5-c413563cd151 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:26.127973 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30665a0f-4d2d-49db-bae5-c413563cd151 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:26.128545 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30665a0f-4d2d-49db-bae5-c413563cd151 admin admin] PATH_INFO: `/v3/projects/2300c43f86074c3b9babc975addbc686/users/4dcfb6941df346f092934f928e5380c8/roles/a6e562cdc96c4be691d638990f8403fd` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:26.134705 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30665a0f-4d2d-49db-bae5-c413563cd151 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2300c43f86074c3b9babc975addbc686, user_id=4dcfb6941df346f092934f928e5380c8, role_id=a6e562cdc96c4be691d638990f8403fd)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:26.147089 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30665a0f-4d2d-49db-bae5-c413563cd151 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:26.179144 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 918/1838] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:34:26 2022] PUT /identity/v3/projects/2300c43f86074c3b9babc975addbc686/users/4dcfb6941df346f092934f928e5380c8/roles/a6e562cdc96c4be691d638990f8403fd => generated 0 bytes in 91 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:34:26.180788 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30665a0f-4d2d-49db-bae5-c413563cd151 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:26.228260 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 921/1839] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:34:25 2022] POST /identity/v3/projects => generated 365 bytes in 238 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:34:26.231398 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4847ea95-2fd5-4c13-8d0d-a4e5714a21ea None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:26.232082 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4847ea95-2fd5-4c13-8d0d-a4e5714a21ea None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:26.232954 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4847ea95-2fd5-4c13-8d0d-a4e5714a21ea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:26.243582 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:34:26.350550 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-4847ea95-2fd5-4c13-8d0d-a4e5714a21ea None None] MFA Rules not processed for user `4dcfb6941df346f092934f928e5380c8`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:34:26.359729 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-4847ea95-2fd5-4c13-8d0d-a4e5714a21ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:26.448279 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30665a0f-4d2d-49db-bae5-c413563cd151 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:26.652797 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-771daadb-8e0e-4097-b64d-a6e5aec7ff7a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:26.654437 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-771daadb-8e0e-4097-b64d-a6e5aec7ff7a admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:26.655756 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-771daadb-8e0e-4097-b64d-a6e5aec7ff7a admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:26.657180 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-771daadb-8e0e-4097-b64d-a6e5aec7ff7a admin admin] PATH_INFO: `/v3/projects/550a5a55876e4f4db25b61677df716fd/users/be1fe9df5bc94ae1926020fb1516f189/roles/02e242d5520740d4b77a5f4008c7eb57` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:26.659907 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 922/1840] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:34:26 2022] POST /identity/v3/auth/tokens => generated 3345 bytes in 431 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:34:26.661449 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4847ea95-2fd5-4c13-8d0d-a4e5714a21ea None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:26.690438 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4847ea95-2fd5-4c13-8d0d-a4e5714a21ea None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:26.719058 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-771daadb-8e0e-4097-b64d-a6e5aec7ff7a admin admin] RBAC: Authorizing `identity:create_grant(project_id=550a5a55876e4f4db25b61677df716fd, user_id=be1fe9df5bc94ae1926020fb1516f189, role_id=02e242d5520740d4b77a5f4008c7eb57)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:26.727632 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55b762db-0dc9-4e72-9d87-60a79c4f19ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:26.732607 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55b762db-0dc9-4e72-9d87-60a79c4f19ba admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:26.735897 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55b762db-0dc9-4e72-9d87-60a79c4f19ba admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:26.736793 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55b762db-0dc9-4e72-9d87-60a79c4f19ba admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:26.745943 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55b762db-0dc9-4e72-9d87-60a79c4f19ba admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:26.748101 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-771daadb-8e0e-4097-b64d-a6e5aec7ff7a admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:26.759115 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55b762db-0dc9-4e72-9d87-60a79c4f19ba admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:26.799877 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 919/1841] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:34:26 2022] PUT /identity/v3/projects/550a5a55876e4f4db25b61677df716fd/users/be1fe9df5bc94ae1926020fb1516f189/roles/02e242d5520740d4b77a5f4008c7eb57 => generated 0 bytes in 620 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:34:26.799877 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-771daadb-8e0e-4097-b64d-a6e5aec7ff7a admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:26.812994 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-771daadb-8e0e-4097-b64d-a6e5aec7ff7a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:26.845527 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 923/1842] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:34:26 2022] POST /identity/v3/users => generated 289 bytes in 184 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:34:26.859276 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76e9d693-d8f7-4afa-ae1d-aff3a98737fc None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:26.859276 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76e9d693-d8f7-4afa-ae1d-aff3a98737fc None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:26.859276 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76e9d693-d8f7-4afa-ae1d-aff3a98737fc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:26.883908 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:34:26.947776 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-76e9d693-d8f7-4afa-ae1d-aff3a98737fc None None] MFA Rules not processed for user `be1fe9df5bc94ae1926020fb1516f189`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:34:26.953129 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-76e9d693-d8f7-4afa-ae1d-aff3a98737fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:27.003001 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-771daadb-8e0e-4097-b64d-a6e5aec7ff7a admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:27.013574 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18d55276-8e70-4b11-8610-1b6445f653c3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dc226f7ddcb49ecacb51ce57fdfc11b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:27.014625 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18d55276-8e70-4b11-8610-1b6445f653c3 service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:27.015016 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18d55276-8e70-4b11-8610-1b6445f653c3 service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:27.015501 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18d55276-8e70-4b11-8610-1b6445f653c3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:27.158215 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 924/1843] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:34:26 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 312 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:34:27.160057 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76e9d693-d8f7-4afa-ae1d-aff3a98737fc None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:27.178750 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:34:27.199764 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18d55276-8e70-4b11-8610-1b6445f653c3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:27.205911 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18d55276-8e70-4b11-8610-1b6445f653c3 service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:27.221986 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 920/1844] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:34:26 2022] GET /identity/v3/auth/tokens => generated 3345 bytes in 421 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:34:27.223584 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18d55276-8e70-4b11-8610-1b6445f653c3 service cinder] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:27.308997 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76e9d693-d8f7-4afa-ae1d-aff3a98737fc None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:27.314615 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0716ef08-2d90-48e2-8ee7-90e50cbf84e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:27.315504 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0716ef08-2d90-48e2-8ee7-90e50cbf84e7 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:27.315937 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0716ef08-2d90-48e2-8ee7-90e50cbf84e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:27.316459 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0716ef08-2d90-48e2-8ee7-90e50cbf84e7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:27.317661 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0716ef08-2d90-48e2-8ee7-90e50cbf84e7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:27.321801 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0716ef08-2d90-48e2-8ee7-90e50cbf84e7 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:27.342227 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 925/1845] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:34:27 2022] GET /identity/v3/roles => generated 1420 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:34:27.344147 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0716ef08-2d90-48e2-8ee7-90e50cbf84e7 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:27.361764 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0716ef08-2d90-48e2-8ee7-90e50cbf84e7 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:27.371852 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fea917de-566b-4275-8ab3-a6f906c70c69 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:27.373262 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fea917de-566b-4275-8ab3-a6f906c70c69 admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:27.373929 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fea917de-566b-4275-8ab3-a6f906c70c69 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:27.374506 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fea917de-566b-4275-8ab3-a6f906c70c69 admin admin] PATH_INFO: `/v3/projects/583bea908b0649de9ca5770b8e716fc7/users/c810b45752d645a78f73b82da50d65cc/roles/a6e562cdc96c4be691d638990f8403fd` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:27.378461 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18d55276-8e70-4b11-8610-1b6445f653c3 service cinder] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:27.404819 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73b1f555-d0d9-4431-bdd6-db0fa0830ac4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:27.408276 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fea917de-566b-4275-8ab3-a6f906c70c69 admin admin] RBAC: Authorizing `identity:create_grant(project_id=583bea908b0649de9ca5770b8e716fc7, user_id=c810b45752d645a78f73b82da50d65cc, role_id=a6e562cdc96c4be691d638990f8403fd)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:27.409702 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73b1f555-d0d9-4431-bdd6-db0fa0830ac4 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:27.411226 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73b1f555-d0d9-4431-bdd6-db0fa0830ac4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:27.412329 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73b1f555-d0d9-4431-bdd6-db0fa0830ac4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:27.415709 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73b1f555-d0d9-4431-bdd6-db0fa0830ac4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:27.416336 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fea917de-566b-4275-8ab3-a6f906c70c69 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:27.420235 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73b1f555-d0d9-4431-bdd6-db0fa0830ac4 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:27.443913 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 926/1846] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:34:27 2022] PUT /identity/v3/projects/583bea908b0649de9ca5770b8e716fc7/users/c810b45752d645a78f73b82da50d65cc/roles/a6e562cdc96c4be691d638990f8403fd => generated 0 bytes in 96 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:34:27.445972 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fea917de-566b-4275-8ab3-a6f906c70c69 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:27.508822 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 921/1847] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:34:27 2022] POST /identity/v3/projects => generated 377 bytes in 286 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:34:27.509980 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73b1f555-d0d9-4431-bdd6-db0fa0830ac4 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:27.627632 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fea917de-566b-4275-8ab3-a6f906c70c69 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:27.649021 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:34:27.662828 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73b1f555-d0d9-4431-bdd6-db0fa0830ac4 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:27.668523 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f581a62-cca8-47b7-a9d6-66cb650d6e22 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dc226f7ddcb49ecacb51ce57fdfc11b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:27.669179 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f581a62-cca8-47b7-a9d6-66cb650d6e22 service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:27.669475 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f581a62-cca8-47b7-a9d6-66cb650d6e22 service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:27.669727 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f581a62-cca8-47b7-a9d6-66cb650d6e22 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:27.778807 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7a58b39-bf34-4a88-bf07-ec067a95b046 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:27.779560 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7a58b39-bf34-4a88-bf07-ec067a95b046 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:27.779933 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7a58b39-bf34-4a88-bf07-ec067a95b046 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:27.780281 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7a58b39-bf34-4a88-bf07-ec067a95b046 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:27.781037 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7a58b39-bf34-4a88-bf07-ec067a95b046 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:27.782776 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7a58b39-bf34-4a88-bf07-ec067a95b046 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:27.783171 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f581a62-cca8-47b7-a9d6-66cb650d6e22 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:27.785149 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f581a62-cca8-47b7-a9d6-66cb650d6e22 service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:27.800298 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 922/1848] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:34:27 2022] GET /identity/v3/auth/tokens => generated 3274 bytes in 291 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:34:27.802209 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f581a62-cca8-47b7-a9d6-66cb650d6e22 service cinder] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:27.819008 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f581a62-cca8-47b7-a9d6-66cb650d6e22 service cinder] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:27.827441 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 927/1849] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:34:27 2022] GET /identity/v3/roles => generated 1420 bytes in 380 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:34:27.827441 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7a58b39-bf34-4a88-bf07-ec067a95b046 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:27.838793 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d897e90-f813-479f-bf0a-00624b038b90 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:27.840230 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d897e90-f813-479f-bf0a-00624b038b90 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:27.840796 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d897e90-f813-479f-bf0a-00624b038b90 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:27.841291 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d897e90-f813-479f-bf0a-00624b038b90 admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:27.851664 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d897e90-f813-479f-bf0a-00624b038b90 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:27.859585 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d897e90-f813-479f-bf0a-00624b038b90 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:27.861112 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7a58b39-bf34-4a88-bf07-ec067a95b046 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:27.888024 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faecfb5d-66ac-431b-a5c0-02fa82b819d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:27.888720 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faecfb5d-66ac-431b-a5c0-02fa82b819d5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:27.889000 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faecfb5d-66ac-431b-a5c0-02fa82b819d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:27.889282 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faecfb5d-66ac-431b-a5c0-02fa82b819d5 admin admin] PATH_INFO: `/v3/projects/583bea908b0649de9ca5770b8e716fc7/users/c810b45752d645a78f73b82da50d65cc/roles/a6e562cdc96c4be691d638990f8403fd` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:27.897086 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faecfb5d-66ac-431b-a5c0-02fa82b819d5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=583bea908b0649de9ca5770b8e716fc7, user_id=c810b45752d645a78f73b82da50d65cc, role_id=a6e562cdc96c4be691d638990f8403fd)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:27.899862 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faecfb5d-66ac-431b-a5c0-02fa82b819d5 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:27.924428 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 928/1850] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:34:27 2022] PUT /identity/v3/projects/583bea908b0649de9ca5770b8e716fc7/users/c810b45752d645a78f73b82da50d65cc/roles/a6e562cdc96c4be691d638990f8403fd => generated 0 bytes in 98 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:34:27.935409 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-696ff2ad-94c0-42e3-80d9-a00dbae14b46 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:27.935722 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-696ff2ad-94c0-42e3-80d9-a00dbae14b46 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:27.935981 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-696ff2ad-94c0-42e3-80d9-a00dbae14b46 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:27.949897 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 923/1851] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:34:27 2022] POST /identity/v3/users => generated 295 bytes in 149 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:34:27.955280 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d897e90-f813-479f-bf0a-00624b038b90 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:28.007970 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d897e90-f813-479f-bf0a-00624b038b90 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:28.039915 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-696ff2ad-94c0-42e3-80d9-a00dbae14b46 None None] MFA Rules not processed for user `c810b45752d645a78f73b82da50d65cc`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:34:28.042014 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-696ff2ad-94c0-42e3-80d9-a00dbae14b46 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:28.046845 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:34:28.226698 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3e3801c-b9be-4238-b6b7-163f90697a81 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:28.227837 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3e3801c-b9be-4238-b6b7-163f90697a81 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:28.228230 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3e3801c-b9be-4238-b6b7-163f90697a81 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:28.228686 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3e3801c-b9be-4238-b6b7-163f90697a81 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:28.229504 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3e3801c-b9be-4238-b6b7-163f90697a81 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:28.233153 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3e3801c-b9be-4238-b6b7-163f90697a81 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:28.256988 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 924/1852] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:34:27 2022] GET /identity/v3/roles => generated 1420 bytes in 301 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:34:28.256988 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3e3801c-b9be-4238-b6b7-163f90697a81 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:28.261386 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-c3e3801c-b9be-4238-b6b7-163f90697a81 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:28.285592 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 929/1853] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:34:27 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 349 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 21:34:28.285736 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3e3801c-b9be-4238-b6b7-163f90697a81 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:28.285858 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-696ff2ad-94c0-42e3-80d9-a00dbae14b46 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:28.300822 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e75d75d5-0f28-48d2-b899-3c1d61260c31 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:28.302244 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e75d75d5-0f28-48d2-b899-3c1d61260c31 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:28.303432 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e75d75d5-0f28-48d2-b899-3c1d61260c31 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:28.304035 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e75d75d5-0f28-48d2-b899-3c1d61260c31 admin admin] PATH_INFO: `/v3/users/704501227d4c4de9821fb2f427aaffad` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:28.305877 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e75d75d5-0f28-48d2-b899-3c1d61260c31 admin admin] RBAC: Authorizing `identity:delete_user(user_id=704501227d4c4de9821fb2f427aaffad)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:28.310396 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e75d75d5-0f28-48d2-b899-3c1d61260c31 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:28.330205 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-696ff2ad-94c0-42e3-80d9-a00dbae14b46 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:28.349725 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9db528df-416e-43be-8df1-c233e805a049 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:28.353220 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db528df-416e-43be-8df1-c233e805a049 admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:28.353730 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db528df-416e-43be-8df1-c233e805a049 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:28.358462 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db528df-416e-43be-8df1-c233e805a049 admin admin] PATH_INFO: `/v3/projects/7d67ea5dfcac44cab179846e256a44e4/users/2100fdb847d34eb796717007b738673e/roles/a6e562cdc96c4be691d638990f8403fd` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:28.403351 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-e75d75d5-0f28-48d2-b899-3c1d61260c31 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '704501227d4c4de9821fb2f427aaffad'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:28.431701 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-e75d75d5-0f28-48d2-b899-3c1d61260c31 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '704501227d4c4de9821fb2f427aaffad'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:28.433894 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-e75d75d5-0f28-48d2-b899-3c1d61260c31 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '704501227d4c4de9821fb2f427aaffad'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:28.435632 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-e75d75d5-0f28-48d2-b899-3c1d61260c31 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '704501227d4c4de9821fb2f427aaffad'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:28.442296 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9db528df-416e-43be-8df1-c233e805a049 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7d67ea5dfcac44cab179846e256a44e4, user_id=2100fdb847d34eb796717007b738673e, role_id=a6e562cdc96c4be691d638990f8403fd)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:28.448466 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-e75d75d5-0f28-48d2-b899-3c1d61260c31 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '704501227d4c4de9821fb2f427aaffad'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:28.458161 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9db528df-416e-43be-8df1-c233e805a049 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:28.486579 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 925/1854] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:34:28 2022] DELETE /identity/v3/users/704501227d4c4de9821fb2f427aaffad => generated 0 bytes in 228 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:34:28.487994 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e75d75d5-0f28-48d2-b899-3c1d61260c31 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:28.490558 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-e75d75d5-0f28-48d2-b899-3c1d61260c31 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:28.533223 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 930/1855] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:34:28 2022] PUT /identity/v3/projects/7d67ea5dfcac44cab179846e256a44e4/users/2100fdb847d34eb796717007b738673e/roles/a6e562cdc96c4be691d638990f8403fd => generated 0 bytes in 249 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:34:28.539417 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93023f04-748e-45dd-bd44-bb0e67e38566 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:28.545616 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93023f04-748e-45dd-bd44-bb0e67e38566 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:28.546062 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93023f04-748e-45dd-bd44-bb0e67e38566 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:28.587547 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-93023f04-748e-45dd-bd44-bb0e67e38566 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:34:28.590262 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-93023f04-748e-45dd-bd44-bb0e67e38566 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:28.700823 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e75d75d5-0f28-48d2-b899-3c1d61260c31 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:28.708617 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:34:28.759563 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 931/1856] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:34:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 226 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:34:28.763141 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93023f04-748e-45dd-bd44-bb0e67e38566 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:28.770902 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-93023f04-748e-45dd-bd44-bb0e67e38566 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:28.821506 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93023f04-748e-45dd-bd44-bb0e67e38566 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:28.840601 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b8742e6-b86f-4fdf-8c79-c7f099efbb4a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:28.849047 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b8742e6-b86f-4fdf-8c79-c7f099efbb4a admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:28.849343 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:34:28.849502 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b8742e6-b86f-4fdf-8c79-c7f099efbb4a admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:28.849502 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b8742e6-b86f-4fdf-8c79-c7f099efbb4a admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:28.858199 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b8742e6-b86f-4fdf-8c79-c7f099efbb4a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:28.861198 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b8742e6-b86f-4fdf-8c79-c7f099efbb4a admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:28.929746 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 932/1857] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:34:28 2022] GET /identity/v3/roles => generated 1420 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:34:28.934878 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b8742e6-b86f-4fdf-8c79-c7f099efbb4a admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:28.937614 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e4c81a0-a7fc-4aaa-b414-dd96c40faf41 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dc226f7ddcb49ecacb51ce57fdfc11b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:28.938480 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-4b8742e6-b86f-4fdf-8c79-c7f099efbb4a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:28.938910 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e4c81a0-a7fc-4aaa-b414-dd96c40faf41 service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:28.939998 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e4c81a0-a7fc-4aaa-b414-dd96c40faf41 service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:28.940352 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e4c81a0-a7fc-4aaa-b414-dd96c40faf41 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:28.948739 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-5e4c81a0-a7fc-4aaa-b414-dd96c40faf41 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:29.092223 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b8742e6-b86f-4fdf-8c79-c7f099efbb4a admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:29.098261 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de532c3a-aab2-4184-814e-0881923c4594 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:29.099796 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de532c3a-aab2-4184-814e-0881923c4594 service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:29.100396 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de532c3a-aab2-4184-814e-0881923c4594 service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:29.101102 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de532c3a-aab2-4184-814e-0881923c4594 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:29.117216 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e4c81a0-a7fc-4aaa-b414-dd96c40faf41 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:29.121612 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e4c81a0-a7fc-4aaa-b414-dd96c40faf41 service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:29.123225 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de532c3a-aab2-4184-814e-0881923c4594 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:29.128326 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de532c3a-aab2-4184-814e-0881923c4594 service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:29.145695 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 926/1858] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:34:28 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 659 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:34:29.150490 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e4c81a0-a7fc-4aaa-b414-dd96c40faf41 service cinder] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:29.158615 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 933/1859] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:34:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 228 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:34:29.175770 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e4c81a0-a7fc-4aaa-b414-dd96c40faf41 service cinder] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:29.186704 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fce054a9-89a7-4a14-bd8a-a44707d384f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:29.187470 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce054a9-89a7-4a14-bd8a-a44707d384f8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:29.187760 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce054a9-89a7-4a14-bd8a-a44707d384f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:29.188041 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce054a9-89a7-4a14-bd8a-a44707d384f8 admin admin] PATH_INFO: `/v3/projects/7d67ea5dfcac44cab179846e256a44e4/users/2100fdb847d34eb796717007b738673e/roles/a6e562cdc96c4be691d638990f8403fd` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:29.191740 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fce054a9-89a7-4a14-bd8a-a44707d384f8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7d67ea5dfcac44cab179846e256a44e4, user_id=2100fdb847d34eb796717007b738673e, role_id=a6e562cdc96c4be691d638990f8403fd)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:29.198452 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fce054a9-89a7-4a14-bd8a-a44707d384f8 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:29.236434 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 927/1860] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:34:29 2022] PUT /identity/v3/projects/7d67ea5dfcac44cab179846e256a44e4/users/2100fdb847d34eb796717007b738673e/roles/a6e562cdc96c4be691d638990f8403fd => generated 0 bytes in 90 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:34:29.247696 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d72dfdf8-8a42-42db-aa2e-777e067eb350 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:29.248183 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d72dfdf8-8a42-42db-aa2e-777e067eb350 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:29.248674 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d72dfdf8-8a42-42db-aa2e-777e067eb350 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:29.331932 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-d72dfdf8-8a42-42db-aa2e-777e067eb350 None None] MFA Rules not processed for user `2100fdb847d34eb796717007b738673e`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:34:29.339353 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-d72dfdf8-8a42-42db-aa2e-777e067eb350 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:29.466146 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 934/1861] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:34:29 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 218 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:34:29.497572 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fce054a9-89a7-4a14-bd8a-a44707d384f8 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:29.501602 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-fce054a9-89a7-4a14-bd8a-a44707d384f8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:29.679899 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fce054a9-89a7-4a14-bd8a-a44707d384f8 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:29.698618 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-340e0009-a3f4-4758-b0f1-dae2b11e2a49 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dc226f7ddcb49ecacb51ce57fdfc11b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:29.699588 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-340e0009-a3f4-4758-b0f1-dae2b11e2a49 service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:29.699876 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-340e0009-a3f4-4758-b0f1-dae2b11e2a49 service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:29.700159 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-340e0009-a3f4-4758-b0f1-dae2b11e2a49 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:29.718172 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-340e0009-a3f4-4758-b0f1-dae2b11e2a49 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:29.718172 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-340e0009-a3f4-4758-b0f1-dae2b11e2a49 service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:29.728810 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 928/1862] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:34:29 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 232 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:34:29.779859 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d72dfdf8-8a42-42db-aa2e-777e067eb350 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:29.782340 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-d72dfdf8-8a42-42db-aa2e-777e067eb350 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:29.816423 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-340e0009-a3f4-4758-b0f1-dae2b11e2a49 service cinder] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:29.844165 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-340e0009-a3f4-4758-b0f1-dae2b11e2a49 service cinder] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:29.851977 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b8b3608-4cdd-4b91-a97a-a56021d94977 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dc226f7ddcb49ecacb51ce57fdfc11b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:29.853463 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b8b3608-4cdd-4b91-a97a-a56021d94977 service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:29.854325 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b8b3608-4cdd-4b91-a97a-a56021d94977 service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:29.854611 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b8b3608-4cdd-4b91-a97a-a56021d94977 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:29.856581 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-9b8b3608-4cdd-4b91-a97a-a56021d94977 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:29.958172 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d72dfdf8-8a42-42db-aa2e-777e067eb350 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:29.971426 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da057331-66da-4621-82dd-eb8b29cac559 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:29.972549 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da057331-66da-4621-82dd-eb8b29cac559 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:29.972929 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da057331-66da-4621-82dd-eb8b29cac559 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:29.973281 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da057331-66da-4621-82dd-eb8b29cac559 admin admin] PATH_INFO: `/v3/projects/9ac4f442da4a45a5b54b5fe400320b9a` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:29.982499 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da057331-66da-4621-82dd-eb8b29cac559 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9ac4f442da4a45a5b54b5fe400320b9a)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:29.984512 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da057331-66da-4621-82dd-eb8b29cac559 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:30.007002 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:34:30.022706 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b8b3608-4cdd-4b91-a97a-a56021d94977 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:30.024803 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b8b3608-4cdd-4b91-a97a-a56021d94977 service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:30.045324 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 929/1863] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:34:29 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 228 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 21:34:30.077730 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-da057331-66da-4621-82dd-eb8b29cac559 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9ac4f442da4a45a5b54b5fe400320b9a', 'request_id': 'req-da057331-66da-4621-82dd-eb8b29cac559'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:30.100121 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-da057331-66da-4621-82dd-eb8b29cac559 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9ac4f442da4a45a5b54b5fe400320b9a', 'request_id': 'req-da057331-66da-4621-82dd-eb8b29cac559'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:30.122659 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-da057331-66da-4621-82dd-eb8b29cac559 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9ac4f442da4a45a5b54b5fe400320b9a', 'request_id': 'req-da057331-66da-4621-82dd-eb8b29cac559'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:30.161470 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-da057331-66da-4621-82dd-eb8b29cac559 admin admin] The token cache is being invalidate because project 9ac4f442da4a45a5b54b5fe400320b9a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81434) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:34:30.161470 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-da057331-66da-4621-82dd-eb8b29cac559 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:30.161470 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-da057331-66da-4621-82dd-eb8b29cac559 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:30.172909 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 935/1864] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:34:29 2022] DELETE /identity/v3/projects/9ac4f442da4a45a5b54b5fe400320b9a => generated 0 bytes in 389 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:34:30.173809 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b8b3608-4cdd-4b91-a97a-a56021d94977 service cinder] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:30.199628 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b8b3608-4cdd-4b91-a97a-a56021d94977 service cinder] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:30.233049 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:34:30.325161 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a3cdf5c-6d5c-42a4-8f3a-1761e96d766e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:30.326104 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a3cdf5c-6d5c-42a4-8f3a-1761e96d766e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:30.326666 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a3cdf5c-6d5c-42a4-8f3a-1761e96d766e admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:30.327344 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a3cdf5c-6d5c-42a4-8f3a-1761e96d766e admin admin] PATH_INFO: `/v3/users/e540d72ab72f4a92923e780ec4574ff8` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:30.329059 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a3cdf5c-6d5c-42a4-8f3a-1761e96d766e admin admin] RBAC: Authorizing `identity:delete_user(user_id=e540d72ab72f4a92923e780ec4574ff8)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:30.333183 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a3cdf5c-6d5c-42a4-8f3a-1761e96d766e admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:30.440117 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-7a3cdf5c-6d5c-42a4-8f3a-1761e96d766e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e540d72ab72f4a92923e780ec4574ff8'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:30.522646 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-7a3cdf5c-6d5c-42a4-8f3a-1761e96d766e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e540d72ab72f4a92923e780ec4574ff8'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:30.524013 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-7a3cdf5c-6d5c-42a4-8f3a-1761e96d766e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e540d72ab72f4a92923e780ec4574ff8'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:30.527501 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-7a3cdf5c-6d5c-42a4-8f3a-1761e96d766e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e540d72ab72f4a92923e780ec4574ff8'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:30.545814 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-7a3cdf5c-6d5c-42a4-8f3a-1761e96d766e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e540d72ab72f4a92923e780ec4574ff8'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:30.588178 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 930/1865] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:34:30 2022] DELETE /identity/v3/users/e540d72ab72f4a92923e780ec4574ff8 => generated 0 bytes in 415 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:34:31.027830 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da057331-66da-4621-82dd-eb8b29cac559 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:31.030713 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-da057331-66da-4621-82dd-eb8b29cac559 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:31.124718 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da057331-66da-4621-82dd-eb8b29cac559 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:31.238352 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bfaefc1-bacf-4648-af42-be01652e0562 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:31.239401 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bfaefc1-bacf-4648-af42-be01652e0562 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:31.240204 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bfaefc1-bacf-4648-af42-be01652e0562 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:31.242345 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bfaefc1-bacf-4648-af42-be01652e0562 admin admin] PATH_INFO: `/v3/users/9110bb38ab0b46818feadb1ab031892c` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:31.244780 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bfaefc1-bacf-4648-af42-be01652e0562 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9110bb38ab0b46818feadb1ab031892c)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:31.249587 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bfaefc1-bacf-4648-af42-be01652e0562 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:31.358715 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-2bfaefc1-bacf-4648-af42-be01652e0562 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9110bb38ab0b46818feadb1ab031892c'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:31.368650 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a3cdf5c-6d5c-42a4-8f3a-1761e96d766e admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:31.374839 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-7a3cdf5c-6d5c-42a4-8f3a-1761e96d766e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:31.385226 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-2bfaefc1-bacf-4648-af42-be01652e0562 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9110bb38ab0b46818feadb1ab031892c'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:31.397770 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-2bfaefc1-bacf-4648-af42-be01652e0562 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9110bb38ab0b46818feadb1ab031892c'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:31.398508 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-2bfaefc1-bacf-4648-af42-be01652e0562 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9110bb38ab0b46818feadb1ab031892c'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:31.399280 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-2bfaefc1-bacf-4648-af42-be01652e0562 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9110bb38ab0b46818feadb1ab031892c'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:31.422119 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a3cdf5c-6d5c-42a4-8f3a-1761e96d766e admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:31.437651 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 936/1866] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:34:31 2022] DELETE /identity/v3/users/9110bb38ab0b46818feadb1ab031892c => generated 0 bytes in 411 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:34:31.445705 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbcbfa58-8b1a-40ae-b105-9ce02879541a None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:31.446179 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbcbfa58-8b1a-40ae-b105-9ce02879541a None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:31.447896 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbcbfa58-8b1a-40ae-b105-9ce02879541a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:31.481959 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-dbcbfa58-8b1a-40ae-b105-9ce02879541a None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:34:31.484846 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-dbcbfa58-8b1a-40ae-b105-9ce02879541a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:31.554493 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:34:31.561588 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7595fe2c-674a-4a4b-8a82-f8dcb7552e18 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:31.562363 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7595fe2c-674a-4a4b-8a82-f8dcb7552e18 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:31.562803 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7595fe2c-674a-4a4b-8a82-f8dcb7552e18 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:31.563091 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7595fe2c-674a-4a4b-8a82-f8dcb7552e18 admin admin] PATH_INFO: `/v3/projects/a24e2a26c6fe46f8a2b7b7306c62c42f` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:31.564971 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7595fe2c-674a-4a4b-8a82-f8dcb7552e18 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a24e2a26c6fe46f8a2b7b7306c62c42f)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:31.567050 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7595fe2c-674a-4a4b-8a82-f8dcb7552e18 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:31.632799 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:34:31.662463 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-7595fe2c-674a-4a4b-8a82-f8dcb7552e18 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a24e2a26c6fe46f8a2b7b7306c62c42f', 'request_id': 'req-7595fe2c-674a-4a4b-8a82-f8dcb7552e18'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:31.674726 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-7595fe2c-674a-4a4b-8a82-f8dcb7552e18 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a24e2a26c6fe46f8a2b7b7306c62c42f', 'request_id': 'req-7595fe2c-674a-4a4b-8a82-f8dcb7552e18'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:31.696261 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 937/1867] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:34:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 253 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:34:31.699529 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-7595fe2c-674a-4a4b-8a82-f8dcb7552e18 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a24e2a26c6fe46f8a2b7b7306c62c42f', 'request_id': 'req-7595fe2c-674a-4a4b-8a82-f8dcb7552e18'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:31.710016 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbcbfa58-8b1a-40ae-b105-9ce02879541a None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:31.713187 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-dbcbfa58-8b1a-40ae-b105-9ce02879541a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:31.744863 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-7595fe2c-674a-4a4b-8a82-f8dcb7552e18 admin admin] The token cache is being invalidate because project a24e2a26c6fe46f8a2b7b7306c62c42f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:34:31.745761 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-7595fe2c-674a-4a4b-8a82-f8dcb7552e18 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:31.749352 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-7595fe2c-674a-4a4b-8a82-f8dcb7552e18 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:31.758332 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 931/1868] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:34:31 2022] DELETE /identity/v3/projects/a24e2a26c6fe46f8a2b7b7306c62c42f => generated 0 bytes in 391 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:34:31.764278 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4f114e5-b086-448c-8cf0-2e173fb0e2fd None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:31.766089 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4f114e5-b086-448c-8cf0-2e173fb0e2fd None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:31.767915 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4f114e5-b086-448c-8cf0-2e173fb0e2fd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:31.852084 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-d4f114e5-b086-448c-8cf0-2e173fb0e2fd None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:34:31.859829 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-d4f114e5-b086-448c-8cf0-2e173fb0e2fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:31.923284 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbcbfa58-8b1a-40ae-b105-9ce02879541a None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:32.073817 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 932/1869] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:34:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 306 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:34:32.073817 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ed4bed5-0980-457f-9f50-22742a25fb66 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:32.073817 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ed4bed5-0980-457f-9f50-22742a25fb66 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:32.073817 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ed4bed5-0980-457f-9f50-22742a25fb66 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:32.153440 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49fb2ab3-08b8-457e-8fe1-c82e0d8faeb8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:32.154384 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49fb2ab3-08b8-457e-8fe1-c82e0d8faeb8 service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:32.154765 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49fb2ab3-08b8-457e-8fe1-c82e0d8faeb8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:32.155151 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49fb2ab3-08b8-457e-8fe1-c82e0d8faeb8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:32.156503 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-8ed4bed5-0980-457f-9f50-22742a25fb66 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:34:32.159064 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-8ed4bed5-0980-457f-9f50-22742a25fb66 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:32.159930 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-49fb2ab3-08b8-457e-8fe1-c82e0d8faeb8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:32.188151 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49fb2ab3-08b8-457e-8fe1-c82e0d8faeb8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:32.190455 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49fb2ab3-08b8-457e-8fe1-c82e0d8faeb8 service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:32.221328 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 938/1870] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:34:31 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 512 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:34:32.251156 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49fb2ab3-08b8-457e-8fe1-c82e0d8faeb8 service neutron] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:32.256100 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 933/1871] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:34:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:34:32.268466 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49fb2ab3-08b8-457e-8fe1-c82e0d8faeb8 service neutron] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:32.275565 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66851ff9-0d3b-415b-a6da-5ccee4dd17d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:32.276322 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66851ff9-0d3b-415b-a6da-5ccee4dd17d9 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:32.276791 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66851ff9-0d3b-415b-a6da-5ccee4dd17d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:32.277161 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66851ff9-0d3b-415b-a6da-5ccee4dd17d9 admin admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:32.278559 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66851ff9-0d3b-415b-a6da-5ccee4dd17d9 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:34:32.278973 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66851ff9-0d3b-415b-a6da-5ccee4dd17d9 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:32.282769 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66851ff9-0d3b-415b-a6da-5ccee4dd17d9 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:32.377210 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 939/1872] 10.222.0.90 () {58 vars in 1153 bytes} [Mon Aug 29 21:34:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 126 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:34:32.377918 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ed4bed5-0980-457f-9f50-22742a25fb66 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:32.413459 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ed4bed5-0980-457f-9f50-22742a25fb66 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:32.429504 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e284f38f-cb66-420d-b0a9-c054a5a7ede5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:32.430272 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e284f38f-cb66-420d-b0a9-c054a5a7ede5 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:32.430671 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e284f38f-cb66-420d-b0a9-c054a5a7ede5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:32.430982 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e284f38f-cb66-420d-b0a9-c054a5a7ede5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:32.432395 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e284f38f-cb66-420d-b0a9-c054a5a7ede5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:32.435676 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e284f38f-cb66-420d-b0a9-c054a5a7ede5 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:32.519915 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66851ff9-0d3b-415b-a6da-5ccee4dd17d9 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:32.537669 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 934/1873] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:34:32 2022] POST /identity/v3/projects => generated 363 bytes in 135 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:34:32.570036 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66851ff9-0d3b-415b-a6da-5ccee4dd17d9 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:32.832887 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8d6eb81-885e-4735-8f50-ff878707e73c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:32.835972 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8d6eb81-885e-4735-8f50-ff878707e73c admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:32.836596 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8d6eb81-885e-4735-8f50-ff878707e73c admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:32.838091 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8d6eb81-885e-4735-8f50-ff878707e73c admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:32.851722 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8d6eb81-885e-4735-8f50-ff878707e73c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:32.861043 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8d6eb81-885e-4735-8f50-ff878707e73c admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:32.895419 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:34:32.908161 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 940/1874] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:34:32 2022] POST /identity/v3/users => generated 282 bytes in 390 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:34:32.916370 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e284f38f-cb66-420d-b0a9-c054a5a7ede5 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:32.995740 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e284f38f-cb66-420d-b0a9-c054a5a7ede5 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:33.021897 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3494c06-f986-4db1-ab10-da93c2a2c32b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:33.022859 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3494c06-f986-4db1-ab10-da93c2a2c32b admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:33.023291 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3494c06-f986-4db1-ab10-da93c2a2c32b admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:33.023708 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3494c06-f986-4db1-ab10-da93c2a2c32b admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:33.024749 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3494c06-f986-4db1-ab10-da93c2a2c32b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:33.027441 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3494c06-f986-4db1-ab10-da93c2a2c32b admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:33.090627 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 935/1875] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:34:32 2022] GET /identity/v3/roles => generated 1420 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:34:33.097287 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8d6eb81-885e-4735-8f50-ff878707e73c admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:33.123907 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8d6eb81-885e-4735-8f50-ff878707e73c admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:33.138715 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38bc753a-8959-4c36-94e9-aa9dbb826292 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:33.140445 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38bc753a-8959-4c36-94e9-aa9dbb826292 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:33.141102 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38bc753a-8959-4c36-94e9-aa9dbb826292 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:33.141816 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38bc753a-8959-4c36-94e9-aa9dbb826292 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:33.142731 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38bc753a-8959-4c36-94e9-aa9dbb826292 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:33.145314 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38bc753a-8959-4c36-94e9-aa9dbb826292 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:33.180430 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3494c06-f986-4db1-ab10-da93c2a2c32b admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:33.194783 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 941/1876] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:34:33 2022] GET /identity/v3/roles => generated 1420 bytes in 97 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:34:33.202973 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3494c06-f986-4db1-ab10-da93c2a2c32b admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:33.221028 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b342e94c-0e5d-4e6a-92a4-72be53b73cbe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:33.222472 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b342e94c-0e5d-4e6a-92a4-72be53b73cbe admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:33.224750 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b342e94c-0e5d-4e6a-92a4-72be53b73cbe admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:33.226726 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b342e94c-0e5d-4e6a-92a4-72be53b73cbe admin admin] PATH_INFO: `/v3/projects/5ee76498fc4747369d7342e932f04c39/users/31bb2e8689d5408b924b2cb7aa98da1c/roles/02e242d5520740d4b77a5f4008c7eb57` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:33.297823 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b342e94c-0e5d-4e6a-92a4-72be53b73cbe admin admin] RBAC: Authorizing `identity:create_grant(project_id=5ee76498fc4747369d7342e932f04c39, user_id=31bb2e8689d5408b924b2cb7aa98da1c, role_id=02e242d5520740d4b77a5f4008c7eb57)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:33.308717 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b342e94c-0e5d-4e6a-92a4-72be53b73cbe admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:33.357926 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 936/1877] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:34:33 2022] PUT /identity/v3/projects/5ee76498fc4747369d7342e932f04c39/users/31bb2e8689d5408b924b2cb7aa98da1c/roles/02e242d5520740d4b77a5f4008c7eb57 => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:34:33.368716 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e6bc2c4-e2e4-4a9c-837d-729af2103e90 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:33.369580 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e6bc2c4-e2e4-4a9c-837d-729af2103e90 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:33.370561 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e6bc2c4-e2e4-4a9c-837d-729af2103e90 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:33.472473 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-8e6bc2c4-e2e4-4a9c-837d-729af2103e90 None None] MFA Rules not processed for user `31bb2e8689d5408b924b2cb7aa98da1c`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:34:33.484084 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-8e6bc2c4-e2e4-4a9c-837d-729af2103e90 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:33.523595 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b342e94c-0e5d-4e6a-92a4-72be53b73cbe admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:33.531799 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-b342e94c-0e5d-4e6a-92a4-72be53b73cbe admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:33.673740 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 942/1878] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:34:33 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 307 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:34:33.680129 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e6bc2c4-e2e4-4a9c-837d-729af2103e90 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:33.686480 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b342e94c-0e5d-4e6a-92a4-72be53b73cbe admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:33.695636 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-836cc990-42d7-478e-bb1d-3c5f97f00082 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:33.696675 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-836cc990-42d7-478e-bb1d-3c5f97f00082 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:33.697037 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-836cc990-42d7-478e-bb1d-3c5f97f00082 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:33.697443 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-836cc990-42d7-478e-bb1d-3c5f97f00082 admin admin] PATH_INFO: `/v3/projects/fa8d6535c8bb407cacdcd3b1acc172cb` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:33.697979 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e6bc2c4-e2e4-4a9c-837d-729af2103e90 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:33.699548 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-836cc990-42d7-478e-bb1d-3c5f97f00082 admin admin] RBAC: Authorizing `identity:delete_project(project_id=fa8d6535c8bb407cacdcd3b1acc172cb)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:33.703528 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-836cc990-42d7-478e-bb1d-3c5f97f00082 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:33.706386 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:DELETE=5,keystone:SELECT=92,keystone:INSERT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:34:33.716971 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f1d67ce-7b04-4b3e-a54a-63cb521b6f0c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:33.717710 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f1d67ce-7b04-4b3e-a54a-63cb521b6f0c admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:33.718135 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f1d67ce-7b04-4b3e-a54a-63cb521b6f0c admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:33.718559 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f1d67ce-7b04-4b3e-a54a-63cb521b6f0c admin admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:33.719299 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f1d67ce-7b04-4b3e-a54a-63cb521b6f0c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:33.725961 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f1d67ce-7b04-4b3e-a54a-63cb521b6f0c admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:33.816688 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 943/1879] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:34:33 2022] POST /identity/v3/projects => generated 363 bytes in 138 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:34:33.824269 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f1d67ce-7b04-4b3e-a54a-63cb521b6f0c admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:33.838210 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-836cc990-42d7-478e-bb1d-3c5f97f00082 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fa8d6535c8bb407cacdcd3b1acc172cb', 'request_id': 'req-836cc990-42d7-478e-bb1d-3c5f97f00082'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:33.874357 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-836cc990-42d7-478e-bb1d-3c5f97f00082 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fa8d6535c8bb407cacdcd3b1acc172cb', 'request_id': 'req-836cc990-42d7-478e-bb1d-3c5f97f00082'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:33.911009 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-836cc990-42d7-478e-bb1d-3c5f97f00082 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fa8d6535c8bb407cacdcd3b1acc172cb', 'request_id': 'req-836cc990-42d7-478e-bb1d-3c5f97f00082'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:33.956795 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-836cc990-42d7-478e-bb1d-3c5f97f00082 admin admin] The token cache is being invalidate because project fa8d6535c8bb407cacdcd3b1acc172cb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:34:33.958026 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-836cc990-42d7-478e-bb1d-3c5f97f00082 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:33.960485 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-836cc990-42d7-478e-bb1d-3c5f97f00082 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:33.966075 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 937/1880] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:34:33 2022] DELETE /identity/v3/projects/fa8d6535c8bb407cacdcd3b1acc172cb => generated 0 bytes in 444 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:34:33.971876 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-836cc990-42d7-478e-bb1d-3c5f97f00082 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:33.978179 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-836cc990-42d7-478e-bb1d-3c5f97f00082 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:34.073006 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f1d67ce-7b04-4b3e-a54a-63cb521b6f0c admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:34.090186 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-783f627b-60a2-4fa4-ac92-b92a92ad5563 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:34.091065 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-783f627b-60a2-4fa4-ac92-b92a92ad5563 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:34.091575 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-783f627b-60a2-4fa4-ac92-b92a92ad5563 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:34.092085 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-783f627b-60a2-4fa4-ac92-b92a92ad5563 admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:34.093178 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-783f627b-60a2-4fa4-ac92-b92a92ad5563 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:34.097263 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-783f627b-60a2-4fa4-ac92-b92a92ad5563 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:34.146772 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 944/1881] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:34:33 2022] POST /identity/v3/users => generated 288 bytes in 323 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:34:34.155236 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-783f627b-60a2-4fa4-ac92-b92a92ad5563 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:34.158395 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-783f627b-60a2-4fa4-ac92-b92a92ad5563 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:34.165728 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-836cc990-42d7-478e-bb1d-3c5f97f00082 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:34.183130 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23e123ac-46b0-487a-93cb-c2fd35537961 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:34.184306 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-783f627b-60a2-4fa4-ac92-b92a92ad5563 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:34.185261 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23e123ac-46b0-487a-93cb-c2fd35537961 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:34.186159 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23e123ac-46b0-487a-93cb-c2fd35537961 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:34.187658 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23e123ac-46b0-487a-93cb-c2fd35537961 admin admin] PATH_INFO: `/v3/users/c810b45752d645a78f73b82da50d65cc` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:34.190572 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23e123ac-46b0-487a-93cb-c2fd35537961 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c810b45752d645a78f73b82da50d65cc)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:34.194301 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7c23e25-9388-43e3-b9cd-9dacffeef6b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:34.195498 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7c23e25-9388-43e3-b9cd-9dacffeef6b6 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:34.196370 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7c23e25-9388-43e3-b9cd-9dacffeef6b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:34.196370 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7c23e25-9388-43e3-b9cd-9dacffeef6b6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:34.197747 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7c23e25-9388-43e3-b9cd-9dacffeef6b6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:34.199586 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23e123ac-46b0-487a-93cb-c2fd35537961 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:34.200871 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7c23e25-9388-43e3-b9cd-9dacffeef6b6 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:34.223421 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 945/1882] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:34:34 2022] GET /identity/v3/roles => generated 1420 bytes in 67 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 29 21:34:34.251003 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7c23e25-9388-43e3-b9cd-9dacffeef6b6 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:34.294777 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7c23e25-9388-43e3-b9cd-9dacffeef6b6 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:34.310260 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-050db9b3-e542-46cf-ac23-380e41f5c356 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:34.311077 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-050db9b3-e542-46cf-ac23-380e41f5c356 admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:34.311359 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-050db9b3-e542-46cf-ac23-380e41f5c356 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:34.311641 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-050db9b3-e542-46cf-ac23-380e41f5c356 admin admin] PATH_INFO: `/v3/projects/03d5e71538dc44bcad7a72876877b7b6/users/7af84e4542ab46978562428f29c8d933/roles/a6e562cdc96c4be691d638990f8403fd` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:34.348085 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-23e123ac-46b0-487a-93cb-c2fd35537961 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c810b45752d645a78f73b82da50d65cc'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:34.357112 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-050db9b3-e542-46cf-ac23-380e41f5c356 admin admin] RBAC: Authorizing `identity:create_grant(project_id=03d5e71538dc44bcad7a72876877b7b6, user_id=7af84e4542ab46978562428f29c8d933, role_id=a6e562cdc96c4be691d638990f8403fd)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:34.361579 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-050db9b3-e542-46cf-ac23-380e41f5c356 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:34.386931 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-23e123ac-46b0-487a-93cb-c2fd35537961 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c810b45752d645a78f73b82da50d65cc'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:34.391109 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-23e123ac-46b0-487a-93cb-c2fd35537961 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c810b45752d645a78f73b82da50d65cc'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:34.393107 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-23e123ac-46b0-487a-93cb-c2fd35537961 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c810b45752d645a78f73b82da50d65cc'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:34.394364 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 946/1883] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:34:34 2022] PUT /identity/v3/projects/03d5e71538dc44bcad7a72876877b7b6/users/7af84e4542ab46978562428f29c8d933/roles/a6e562cdc96c4be691d638990f8403fd => generated 0 bytes in 138 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:34:34.396273 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-050db9b3-e542-46cf-ac23-380e41f5c356 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:34.407188 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-050db9b3-e542-46cf-ac23-380e41f5c356 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:34.416940 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-23e123ac-46b0-487a-93cb-c2fd35537961 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c810b45752d645a78f73b82da50d65cc'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:34.433082 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:34:34.455662 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 938/1884] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:34:33 2022] DELETE /identity/v3/users/c810b45752d645a78f73b82da50d65cc => generated 0 bytes in 484 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:34:34.457473 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23e123ac-46b0-487a-93cb-c2fd35537961 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:34.478420 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-23e123ac-46b0-487a-93cb-c2fd35537961 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:34.541823 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:34:34.602296 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-050db9b3-e542-46cf-ac23-380e41f5c356 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:34.758663 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23e123ac-46b0-487a-93cb-c2fd35537961 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:34.776846 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35323aaa-8420-4077-a612-521a46cf36ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:34.778849 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35323aaa-8420-4077-a612-521a46cf36ca admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:34.780007 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35323aaa-8420-4077-a612-521a46cf36ca admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:34.780742 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35323aaa-8420-4077-a612-521a46cf36ca admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:34.785072 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35323aaa-8420-4077-a612-521a46cf36ca admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:34.792853 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35323aaa-8420-4077-a612-521a46cf36ca admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:34.822308 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-190e1fdb-f40a-48cc-b228-6f76a080d9e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:34.823312 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-190e1fdb-f40a-48cc-b228-6f76a080d9e9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:34.823759 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-190e1fdb-f40a-48cc-b228-6f76a080d9e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:34.824498 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-190e1fdb-f40a-48cc-b228-6f76a080d9e9 admin admin] PATH_INFO: `/v3/users/8a9f06d9417f4e919aa63807fb7890aa` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:34.829273 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-190e1fdb-f40a-48cc-b228-6f76a080d9e9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8a9f06d9417f4e919aa63807fb7890aa)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:34.834223 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 939/1885] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:34:34 2022] GET /identity/v3/roles => generated 1420 bytes in 378 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:34:34.836095 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-190e1fdb-f40a-48cc-b228-6f76a080d9e9 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:34.842006 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35323aaa-8420-4077-a612-521a46cf36ca admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:34.883911 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35323aaa-8420-4077-a612-521a46cf36ca admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:34.903642 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ada7241-9422-448a-b5b3-cdeb94cd44ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:34.904709 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ada7241-9422-448a-b5b3-cdeb94cd44ce admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:34.906445 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ada7241-9422-448a-b5b3-cdeb94cd44ce admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:34.909778 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ada7241-9422-448a-b5b3-cdeb94cd44ce admin admin] PATH_INFO: `/v3/projects/03d5e71538dc44bcad7a72876877b7b6/users/7af84e4542ab46978562428f29c8d933/roles/a6e562cdc96c4be691d638990f8403fd` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:34.919131 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ada7241-9422-448a-b5b3-cdeb94cd44ce admin admin] RBAC: Authorizing `identity:create_grant(project_id=03d5e71538dc44bcad7a72876877b7b6, user_id=7af84e4542ab46978562428f29c8d933, role_id=a6e562cdc96c4be691d638990f8403fd)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:34.939443 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ada7241-9422-448a-b5b3-cdeb94cd44ce admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:34.987433 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 940/1886] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:34:34 2022] PUT /identity/v3/projects/03d5e71538dc44bcad7a72876877b7b6/users/7af84e4542ab46978562428f29c8d933/roles/a6e562cdc96c4be691d638990f8403fd => generated 0 bytes in 148 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 21:34:35.006331 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bc52600-df29-48b3-b566-4b2182c352c0 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:35.006897 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bc52600-df29-48b3-b566-4b2182c352c0 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:35.007375 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bc52600-df29-48b3-b566-4b2182c352c0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:35.023102 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-190e1fdb-f40a-48cc-b228-6f76a080d9e9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8a9f06d9417f4e919aa63807fb7890aa'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:35.039745 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-190e1fdb-f40a-48cc-b228-6f76a080d9e9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8a9f06d9417f4e919aa63807fb7890aa'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:35.057453 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-190e1fdb-f40a-48cc-b228-6f76a080d9e9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8a9f06d9417f4e919aa63807fb7890aa'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:35.058521 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-190e1fdb-f40a-48cc-b228-6f76a080d9e9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8a9f06d9417f4e919aa63807fb7890aa'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:35.059165 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-190e1fdb-f40a-48cc-b228-6f76a080d9e9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8a9f06d9417f4e919aa63807fb7890aa'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:35.127384 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-1bc52600-df29-48b3-b566-4b2182c352c0 None None] MFA Rules not processed for user `7af84e4542ab46978562428f29c8d933`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:34:35.128999 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-1bc52600-df29-48b3-b566-4b2182c352c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:35.155864 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 947/1887] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:34:34 2022] DELETE /identity/v3/users/8a9f06d9417f4e919aa63807fb7890aa => generated 0 bytes in 761 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:34:35.157312 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-190e1fdb-f40a-48cc-b228-6f76a080d9e9 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:35.161077 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-190e1fdb-f40a-48cc-b228-6f76a080d9e9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:35.289244 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:34:35.362743 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-190e1fdb-f40a-48cc-b228-6f76a080d9e9 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:35.385101 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df3443bd-bd0b-40fe-9c59-cc1ee99a9a76 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:35.386394 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df3443bd-bd0b-40fe-9c59-cc1ee99a9a76 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:35.386868 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df3443bd-bd0b-40fe-9c59-cc1ee99a9a76 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:35.388710 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df3443bd-bd0b-40fe-9c59-cc1ee99a9a76 admin admin] PATH_INFO: `/v3/users/518273de70ab4b5c9bda2fcb541610bd` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:35.393310 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df3443bd-bd0b-40fe-9c59-cc1ee99a9a76 admin admin] RBAC: Authorizing `identity:delete_user(user_id=518273de70ab4b5c9bda2fcb541610bd)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:35.400460 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df3443bd-bd0b-40fe-9c59-cc1ee99a9a76 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:35.409928 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 941/1888] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:34:35 2022] POST /identity/v3/auth/tokens => generated 3307 bytes in 406 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:34:35.411154 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adcf88e4-6afe-4ef9-92c1-0d326e77a9ca None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:35.411504 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adcf88e4-6afe-4ef9-92c1-0d326e77a9ca None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:35.412277 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adcf88e4-6afe-4ef9-92c1-0d326e77a9ca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:35.457554 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-adcf88e4-6afe-4ef9-92c1-0d326e77a9ca None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:34:35.464701 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-adcf88e4-6afe-4ef9-92c1-0d326e77a9ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:35.484988 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 942/1889] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:34:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 68 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:34:35.486934 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adcf88e4-6afe-4ef9-92c1-0d326e77a9ca None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:35.493491 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-adcf88e4-6afe-4ef9-92c1-0d326e77a9ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:35.538899 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-df3443bd-bd0b-40fe-9c59-cc1ee99a9a76 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '518273de70ab4b5c9bda2fcb541610bd'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:35.549450 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-df3443bd-bd0b-40fe-9c59-cc1ee99a9a76 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '518273de70ab4b5c9bda2fcb541610bd'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:35.563715 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-df3443bd-bd0b-40fe-9c59-cc1ee99a9a76 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '518273de70ab4b5c9bda2fcb541610bd'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:35.564603 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-df3443bd-bd0b-40fe-9c59-cc1ee99a9a76 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '518273de70ab4b5c9bda2fcb541610bd'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:35.565766 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-df3443bd-bd0b-40fe-9c59-cc1ee99a9a76 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '518273de70ab4b5c9bda2fcb541610bd'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:35.598068 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 948/1890] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:34:35 2022] DELETE /identity/v3/users/518273de70ab4b5c9bda2fcb541610bd => generated 0 bytes in 442 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:34:35.600364 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df3443bd-bd0b-40fe-9c59-cc1ee99a9a76 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:35.603218 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-df3443bd-bd0b-40fe-9c59-cc1ee99a9a76 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:35.604520 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adcf88e4-6afe-4ef9-92c1-0d326e77a9ca None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:35.663100 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:34:35.715439 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a7bf154-cdab-4a62-b140-e65393953e90 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dc226f7ddcb49ecacb51ce57fdfc11b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:35.716587 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a7bf154-cdab-4a62-b140-e65393953e90 service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:35.717034 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df3443bd-bd0b-40fe-9c59-cc1ee99a9a76 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:35.717148 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a7bf154-cdab-4a62-b140-e65393953e90 service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:35.717501 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a7bf154-cdab-4a62-b140-e65393953e90 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:35.720310 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-6a7bf154-cdab-4a62-b140-e65393953e90 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:35.722787 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-421f34bd-9873-4ff1-8f95-f80f114c2b56 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:35.723949 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-421f34bd-9873-4ff1-8f95-f80f114c2b56 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:35.724232 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-421f34bd-9873-4ff1-8f95-f80f114c2b56 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:35.724512 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-421f34bd-9873-4ff1-8f95-f80f114c2b56 admin admin] PATH_INFO: `/v3/projects/583bea908b0649de9ca5770b8e716fc7` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:35.725797 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-421f34bd-9873-4ff1-8f95-f80f114c2b56 admin admin] RBAC: Authorizing `identity:delete_project(project_id=583bea908b0649de9ca5770b8e716fc7)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:35.727964 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-421f34bd-9873-4ff1-8f95-f80f114c2b56 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:35.803497 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-421f34bd-9873-4ff1-8f95-f80f114c2b56 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '583bea908b0649de9ca5770b8e716fc7', 'request_id': 'req-421f34bd-9873-4ff1-8f95-f80f114c2b56'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:35.814913 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=83,keystone:DELETE=16,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:34:35.821394 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a7bf154-cdab-4a62-b140-e65393953e90 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:35.821756 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-421f34bd-9873-4ff1-8f95-f80f114c2b56 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '583bea908b0649de9ca5770b8e716fc7', 'request_id': 'req-421f34bd-9873-4ff1-8f95-f80f114c2b56'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:35.823001 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a7bf154-cdab-4a62-b140-e65393953e90 service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:35.831853 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-421f34bd-9873-4ff1-8f95-f80f114c2b56 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '583bea908b0649de9ca5770b8e716fc7', 'request_id': 'req-421f34bd-9873-4ff1-8f95-f80f114c2b56'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:35.837295 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 943/1891] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:34:35 2022] GET /identity/v3/auth/tokens => generated 3307 bytes in 352 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:34:35.838214 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a7bf154-cdab-4a62-b140-e65393953e90 service cinder] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:35.839939 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-6a7bf154-cdab-4a62-b140-e65393953e90 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:35.882820 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-421f34bd-9873-4ff1-8f95-f80f114c2b56 admin admin] The token cache is being invalidate because project 583bea908b0649de9ca5770b8e716fc7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81434) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:34:35.883529 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-421f34bd-9873-4ff1-8f95-f80f114c2b56 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:35.885899 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-421f34bd-9873-4ff1-8f95-f80f114c2b56 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:35.887931 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 949/1892] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:34:35 2022] DELETE /identity/v3/projects/583bea908b0649de9ca5770b8e716fc7 => generated 0 bytes in 289 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:34:35.888937 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-417b889f-c428-49f8-a131-9b65d39e478e None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:35.889226 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-417b889f-c428-49f8-a131-9b65d39e478e None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:35.889505 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-417b889f-c428-49f8-a131-9b65d39e478e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:35.926255 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-417b889f-c428-49f8-a131-9b65d39e478e None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:34:35.928997 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-417b889f-c428-49f8-a131-9b65d39e478e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:35.999756 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a7bf154-cdab-4a62-b140-e65393953e90 service cinder] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:36.044487 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 950/1893] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:34:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:34:36.046065 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c44eefd4-2199-464c-959e-2cb6306006f1 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:36.046449 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c44eefd4-2199-464c-959e-2cb6306006f1 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:36.046866 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c44eefd4-2199-464c-959e-2cb6306006f1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:36.074509 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-c44eefd4-2199-464c-959e-2cb6306006f1 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:34:36.075892 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-c44eefd4-2199-464c-959e-2cb6306006f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:36.078590 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b6a7e0c-c4c2-4ea4-ba40-0866c1b5ab98 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:36.079346 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b6a7e0c-c4c2-4ea4-ba40-0866c1b5ab98 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:36.079727 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b6a7e0c-c4c2-4ea4-ba40-0866c1b5ab98 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:36.080097 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b6a7e0c-c4c2-4ea4-ba40-0866c1b5ab98 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:36.082474 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-7b6a7e0c-c4c2-4ea4-ba40-0866c1b5ab98 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:36.090933 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 951/1894] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:34:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 45 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:34:36.094392 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c44eefd4-2199-464c-959e-2cb6306006f1 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:36.102916 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-c44eefd4-2199-464c-959e-2cb6306006f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:36.118041 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b6a7e0c-c4c2-4ea4-ba40-0866c1b5ab98 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:36.120175 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b6a7e0c-c4c2-4ea4-ba40-0866c1b5ab98 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:36.133692 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 944/1895] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:34:35 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 296 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:34:36.135031 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b6a7e0c-c4c2-4ea4-ba40-0866c1b5ab98 service neutron] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:36.136901 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-7b6a7e0c-c4c2-4ea4-ba40-0866c1b5ab98 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:36.151689 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b6a7e0c-c4c2-4ea4-ba40-0866c1b5ab98 service neutron] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:36.156511 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f819c068-aaaa-4f96-bcec-78a8ee20562c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:36.157473 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f819c068-aaaa-4f96-bcec-78a8ee20562c service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:36.158388 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f819c068-aaaa-4f96-bcec-78a8ee20562c service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:36.158388 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f819c068-aaaa-4f96-bcec-78a8ee20562c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:36.172025 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f819c068-aaaa-4f96-bcec-78a8ee20562c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:36.173697 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f819c068-aaaa-4f96-bcec-78a8ee20562c service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:36.191494 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 945/1896] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:34:36 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 57 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:34:36.194343 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e1e3711-a972-45c0-bdfc-25f584044667 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:36.195104 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e1e3711-a972-45c0-bdfc-25f584044667 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:36.195713 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e1e3711-a972-45c0-bdfc-25f584044667 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:36.220395 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c44eefd4-2199-464c-959e-2cb6306006f1 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:36.225773 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98c4a026-1f12-44d2-9887-f3868c3c1f6e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dc226f7ddcb49ecacb51ce57fdfc11b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:36.226468 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98c4a026-1f12-44d2-9887-f3868c3c1f6e service cinder] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:36.226798 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98c4a026-1f12-44d2-9887-f3868c3c1f6e service cinder] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:36.227102 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98c4a026-1f12-44d2-9887-f3868c3c1f6e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:36.229589 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-8e1e3711-a972-45c0-bdfc-25f584044667 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:34:36.230634 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-98c4a026-1f12-44d2-9887-f3868c3c1f6e service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:36.231900 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-8e1e3711-a972-45c0-bdfc-25f584044667 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:36.245158 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 946/1897] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:34:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 50 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:34:36.248138 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e1e3711-a972-45c0-bdfc-25f584044667 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:36.277445 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e1e3711-a972-45c0-bdfc-25f584044667 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:36.290155 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-523f8103-7606-4451-86a2-604e5e573caa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:36.294637 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-523f8103-7606-4451-86a2-604e5e573caa admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:36.295730 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-523f8103-7606-4451-86a2-604e5e573caa admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:36.299443 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-523f8103-7606-4451-86a2-604e5e573caa admin admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:36.300817 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-523f8103-7606-4451-86a2-604e5e573caa admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:34:36.301433 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-523f8103-7606-4451-86a2-604e5e573caa admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:36.307766 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-523f8103-7606-4451-86a2-604e5e573caa admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:36.359159 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 947/1898] 10.222.0.90 () {58 vars in 1153 bytes} [Mon Aug 29 21:34:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 111 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:34:36.363645 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-523f8103-7606-4451-86a2-604e5e573caa admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:36.372650 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:34:36.397343 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-523f8103-7606-4451-86a2-604e5e573caa admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:36.408567 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e86d0d16-7a8e-4b46-9f23-9f002e223d35 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:36.409370 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e86d0d16-7a8e-4b46-9f23-9f002e223d35 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:36.411520 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e86d0d16-7a8e-4b46-9f23-9f002e223d35 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:36.411962 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e86d0d16-7a8e-4b46-9f23-9f002e223d35 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:36.413122 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e86d0d16-7a8e-4b46-9f23-9f002e223d35 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:36.415556 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e86d0d16-7a8e-4b46-9f23-9f002e223d35 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:36.500314 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98c4a026-1f12-44d2-9887-f3868c3c1f6e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:36.508677 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98c4a026-1f12-44d2-9887-f3868c3c1f6e service cinder] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:36.536070 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 948/1899] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:34:36 2022] POST /identity/v3/projects => generated 367 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 21:34:36.543990 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e86d0d16-7a8e-4b46-9f23-9f002e223d35 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:36.558219 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 952/1900] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:34:36 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 462 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:34:36.682097 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e86d0d16-7a8e-4b46-9f23-9f002e223d35 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:36.695000 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f374c6f-175f-4112-917b-02d8aa4bb43a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:36.696242 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f374c6f-175f-4112-917b-02d8aa4bb43a admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:36.696806 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f374c6f-175f-4112-917b-02d8aa4bb43a admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:36.697298 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f374c6f-175f-4112-917b-02d8aa4bb43a admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:36.700184 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f374c6f-175f-4112-917b-02d8aa4bb43a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:36.704738 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f374c6f-175f-4112-917b-02d8aa4bb43a admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:36.758147 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 949/1901] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:34:36 2022] POST /identity/v3/users => generated 284 bytes in 215 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 21:34:36.758922 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98c4a026-1f12-44d2-9887-f3868c3c1f6e service cinder] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:36.778246 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98c4a026-1f12-44d2-9887-f3868c3c1f6e service cinder] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:36.791327 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94fa6fb3-615d-497b-9e08-5cec30579d84 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:36.793936 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94fa6fb3-615d-497b-9e08-5cec30579d84 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:36.796552 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94fa6fb3-615d-497b-9e08-5cec30579d84 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:36.797060 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94fa6fb3-615d-497b-9e08-5cec30579d84 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:36.798907 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94fa6fb3-615d-497b-9e08-5cec30579d84 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:36.803192 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94fa6fb3-615d-497b-9e08-5cec30579d84 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:36.847139 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f374c6f-175f-4112-917b-02d8aa4bb43a admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:36.855586 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 953/1902] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:34:36 2022] GET /identity/v3/roles => generated 1420 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:34:36.923015 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f374c6f-175f-4112-917b-02d8aa4bb43a admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:36.931926 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94fa6fb3-615d-497b-9e08-5cec30579d84 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:36.933871 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-94fa6fb3-615d-497b-9e08-5cec30579d84 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:36.949370 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6884682-61eb-4010-ba98-ff5a76d8d30c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:36.950004 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6884682-61eb-4010-ba98-ff5a76d8d30c admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:36.950259 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6884682-61eb-4010-ba98-ff5a76d8d30c admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:36.950543 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6884682-61eb-4010-ba98-ff5a76d8d30c admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:36.951180 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6884682-61eb-4010-ba98-ff5a76d8d30c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:36.952812 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6884682-61eb-4010-ba98-ff5a76d8d30c admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:36.966610 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94fa6fb3-615d-497b-9e08-5cec30579d84 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:37.001334 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0db86c3a-b4fb-403f-95b5-416fd04b6aa5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:37.003537 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 950/1903] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:34:36 2022] GET /identity/v3/roles => generated 1420 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:34:37.009875 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db86c3a-b4fb-403f-95b5-416fd04b6aa5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:37.010222 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6884682-61eb-4010-ba98-ff5a76d8d30c admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:37.011191 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db86c3a-b4fb-403f-95b5-416fd04b6aa5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:37.011714 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db86c3a-b4fb-403f-95b5-416fd04b6aa5 admin admin] PATH_INFO: `/v3/projects/78a47c9c29634eeea2f1675ee555bb04` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:37.019557 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0db86c3a-b4fb-403f-95b5-416fd04b6aa5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=78a47c9c29634eeea2f1675ee555bb04)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:37.024168 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0db86c3a-b4fb-403f-95b5-416fd04b6aa5 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:37.033260 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6884682-61eb-4010-ba98-ff5a76d8d30c admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:37.040590 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3ba4648-a01e-4253-b5ed-84c9f6cf0fea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:37.041180 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3ba4648-a01e-4253-b5ed-84c9f6cf0fea admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:37.041567 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3ba4648-a01e-4253-b5ed-84c9f6cf0fea admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:37.041730 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3ba4648-a01e-4253-b5ed-84c9f6cf0fea admin admin] PATH_INFO: `/v3/projects/c04c6210dab04f3d9a6cf79517956a25/users/858dff9823754bae941f3cc2e957052c/roles/02e242d5520740d4b77a5f4008c7eb57` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:37.076952 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3ba4648-a01e-4253-b5ed-84c9f6cf0fea admin admin] RBAC: Authorizing `identity:create_grant(project_id=c04c6210dab04f3d9a6cf79517956a25, user_id=858dff9823754bae941f3cc2e957052c, role_id=02e242d5520740d4b77a5f4008c7eb57)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:37.081441 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3ba4648-a01e-4253-b5ed-84c9f6cf0fea admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:37.089730 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=98,keystone:INSERT=1,keystone:DELETE=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:34:37.109971 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 951/1904] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:34:37 2022] PUT /identity/v3/projects/c04c6210dab04f3d9a6cf79517956a25/users/858dff9823754bae941f3cc2e957052c/roles/02e242d5520740d4b77a5f4008c7eb57 => generated 0 bytes in 95 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:34:37.120756 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9793b2f1-e136-47bd-9f7e-9f5c3dda6400 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:37.121314 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9793b2f1-e136-47bd-9f7e-9f5c3dda6400 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:37.123120 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9793b2f1-e136-47bd-9f7e-9f5c3dda6400 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:37.178142 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-0db86c3a-b4fb-403f-95b5-416fd04b6aa5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '78a47c9c29634eeea2f1675ee555bb04', 'request_id': 'req-0db86c3a-b4fb-403f-95b5-416fd04b6aa5'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:37.185418 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-0db86c3a-b4fb-403f-95b5-416fd04b6aa5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '78a47c9c29634eeea2f1675ee555bb04', 'request_id': 'req-0db86c3a-b4fb-403f-95b5-416fd04b6aa5'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:37.195329 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-0db86c3a-b4fb-403f-95b5-416fd04b6aa5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '78a47c9c29634eeea2f1675ee555bb04', 'request_id': 'req-0db86c3a-b4fb-403f-95b5-416fd04b6aa5'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:37.196864 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-9793b2f1-e136-47bd-9f7e-9f5c3dda6400 None None] MFA Rules not processed for user `858dff9823754bae941f3cc2e957052c`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:34:37.199619 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-9793b2f1-e136-47bd-9f7e-9f5c3dda6400 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:37.216026 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-0db86c3a-b4fb-403f-95b5-416fd04b6aa5 admin admin] The token cache is being invalidate because project 78a47c9c29634eeea2f1675ee555bb04 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81434) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:34:37.216560 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-0db86c3a-b4fb-403f-95b5-416fd04b6aa5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:37.217599 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-0db86c3a-b4fb-403f-95b5-416fd04b6aa5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:37.226758 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 954/1905] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:34:36 2022] DELETE /identity/v3/projects/78a47c9c29634eeea2f1675ee555bb04 => generated 0 bytes in 290 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 21:34:37.229855 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0db86c3a-b4fb-403f-95b5-416fd04b6aa5 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:37.232321 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-0db86c3a-b4fb-403f-95b5-416fd04b6aa5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:37.372120 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 952/1906] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:34:37 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 253 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:34:37.380680 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9793b2f1-e136-47bd-9f7e-9f5c3dda6400 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:37.389542 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-9793b2f1-e136-47bd-9f7e-9f5c3dda6400 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:37.426143 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0db86c3a-b4fb-403f-95b5-416fd04b6aa5 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:37.442806 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8810541-11df-453f-8f71-46760bd53f74 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:37.444084 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8810541-11df-453f-8f71-46760bd53f74 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:37.444572 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8810541-11df-453f-8f71-46760bd53f74 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:37.445056 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8810541-11df-453f-8f71-46760bd53f74 admin admin] PATH_INFO: `/v3/users/be1fe9df5bc94ae1926020fb1516f189` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:37.447192 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8810541-11df-453f-8f71-46760bd53f74 admin admin] RBAC: Authorizing `identity:delete_user(user_id=be1fe9df5bc94ae1926020fb1516f189)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:37.450306 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:34:37.450851 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8810541-11df-453f-8f71-46760bd53f74 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:37.559016 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-a8810541-11df-453f-8f71-46760bd53f74 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'be1fe9df5bc94ae1926020fb1516f189'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:37.570057 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9793b2f1-e136-47bd-9f7e-9f5c3dda6400 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:37.573579 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-a8810541-11df-453f-8f71-46760bd53f74 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'be1fe9df5bc94ae1926020fb1516f189'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:37.594093 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8c652cc-e49e-4627-a77a-c997e7aac9a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:37.594093 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8c652cc-e49e-4627-a77a-c997e7aac9a5 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:37.594093 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8c652cc-e49e-4627-a77a-c997e7aac9a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:37.594093 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8c652cc-e49e-4627-a77a-c997e7aac9a5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:37.594850 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8c652cc-e49e-4627-a77a-c997e7aac9a5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:37.598674 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8c652cc-e49e-4627-a77a-c997e7aac9a5 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:37.605019 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-a8810541-11df-453f-8f71-46760bd53f74 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'be1fe9df5bc94ae1926020fb1516f189'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:37.605762 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-a8810541-11df-453f-8f71-46760bd53f74 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'be1fe9df5bc94ae1926020fb1516f189'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:37.607655 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-a8810541-11df-453f-8f71-46760bd53f74 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'be1fe9df5bc94ae1926020fb1516f189'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:37.641151 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 955/1907] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:34:37 2022] DELETE /identity/v3/users/be1fe9df5bc94ae1926020fb1516f189 => generated 0 bytes in 413 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:34:37.642471 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8810541-11df-453f-8f71-46760bd53f74 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:37.655978 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-a8810541-11df-453f-8f71-46760bd53f74 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:37.714957 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 953/1908] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:34:37 2022] POST /identity/v3/projects => generated 369 bytes in 338 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:34:37.719396 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8c652cc-e49e-4627-a77a-c997e7aac9a5 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:37.729174 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-f8c652cc-e49e-4627-a77a-c997e7aac9a5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:37.791124 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:34:37.883576 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8c652cc-e49e-4627-a77a-c997e7aac9a5 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:37.891580 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37b78203-e3c0-40d1-a01c-520a65a00502 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:37.892208 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b78203-e3c0-40d1-a01c-520a65a00502 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:37.893254 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b78203-e3c0-40d1-a01c-520a65a00502 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:37.893555 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b78203-e3c0-40d1-a01c-520a65a00502 admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:37.894297 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37b78203-e3c0-40d1-a01c-520a65a00502 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:37.899535 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37b78203-e3c0-40d1-a01c-520a65a00502 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:37.925724 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8810541-11df-453f-8f71-46760bd53f74 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:37.953358 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 954/1909] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:34:37 2022] POST /identity/v3/users => generated 291 bytes in 230 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:34:37.953590 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fc1de6b-5c09-4bf5-a360-96ec6f1d652c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:37.955658 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fc1de6b-5c09-4bf5-a360-96ec6f1d652c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:37.956038 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fc1de6b-5c09-4bf5-a360-96ec6f1d652c admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:37.956254 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37b78203-e3c0-40d1-a01c-520a65a00502 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:37.956693 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fc1de6b-5c09-4bf5-a360-96ec6f1d652c admin admin] PATH_INFO: `/v3/projects/2377a7c29f31404cb421097da62aae7b` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:37.962774 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fc1de6b-5c09-4bf5-a360-96ec6f1d652c admin admin] RBAC: Authorizing `identity:delete_project(project_id=2377a7c29f31404cb421097da62aae7b)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:37.965194 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fc1de6b-5c09-4bf5-a360-96ec6f1d652c admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:37.980743 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37b78203-e3c0-40d1-a01c-520a65a00502 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:38.015577 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-838230d7-dc05-4f89-bba8-533006c61ded admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:38.016244 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-838230d7-dc05-4f89-bba8-533006c61ded admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:38.018908 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-838230d7-dc05-4f89-bba8-533006c61ded admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:38.019221 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-838230d7-dc05-4f89-bba8-533006c61ded admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:38.023285 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-838230d7-dc05-4f89-bba8-533006c61ded admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:38.028029 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-838230d7-dc05-4f89-bba8-533006c61ded admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:38.109976 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 955/1910] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:34:37 2022] GET /identity/v3/roles => generated 1420 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:34:38.112491 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-838230d7-dc05-4f89-bba8-533006c61ded admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:38.150968 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-4fc1de6b-5c09-4bf5-a360-96ec6f1d652c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2377a7c29f31404cb421097da62aae7b', 'request_id': 'req-4fc1de6b-5c09-4bf5-a360-96ec6f1d652c'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:38.152671 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-838230d7-dc05-4f89-bba8-533006c61ded admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:38.165123 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-4fc1de6b-5c09-4bf5-a360-96ec6f1d652c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2377a7c29f31404cb421097da62aae7b', 'request_id': 'req-4fc1de6b-5c09-4bf5-a360-96ec6f1d652c'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:38.181016 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-4fc1de6b-5c09-4bf5-a360-96ec6f1d652c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2377a7c29f31404cb421097da62aae7b', 'request_id': 'req-4fc1de6b-5c09-4bf5-a360-96ec6f1d652c'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:38.231361 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-4fc1de6b-5c09-4bf5-a360-96ec6f1d652c admin admin] The token cache is being invalidate because project 2377a7c29f31404cb421097da62aae7b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81434) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:34:38.231735 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-4fc1de6b-5c09-4bf5-a360-96ec6f1d652c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:38.234286 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-4fc1de6b-5c09-4bf5-a360-96ec6f1d652c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:38.237863 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 956/1911] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:34:37 2022] DELETE /identity/v3/projects/2377a7c29f31404cb421097da62aae7b => generated 0 bytes in 596 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:34:38.246597 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fc1de6b-5c09-4bf5-a360-96ec6f1d652c admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:38.250878 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-4fc1de6b-5c09-4bf5-a360-96ec6f1d652c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:38.341192 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca8be285-1902-403a-b50b-e71a3e12db5d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:38.348421 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca8be285-1902-403a-b50b-e71a3e12db5d admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:38.356074 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca8be285-1902-403a-b50b-e71a3e12db5d admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:38.356793 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca8be285-1902-403a-b50b-e71a3e12db5d admin admin] PATH_INFO: `/v3/projects/3fa41cc81c4147c287a44d7cefa6e77b/users/f91680f344214808954493dc7e276a5f/roles/a6e562cdc96c4be691d638990f8403fd` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:38.390711 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fc1de6b-5c09-4bf5-a360-96ec6f1d652c admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:38.399290 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcc91be6-98d0-46c5-983b-5ba6b99b74d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:38.400148 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcc91be6-98d0-46c5-983b-5ba6b99b74d9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:38.401304 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcc91be6-98d0-46c5-983b-5ba6b99b74d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:38.401304 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcc91be6-98d0-46c5-983b-5ba6b99b74d9 admin admin] PATH_INFO: `/v3/users/4dcfb6941df346f092934f928e5380c8` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:38.402694 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcc91be6-98d0-46c5-983b-5ba6b99b74d9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4dcfb6941df346f092934f928e5380c8)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:38.405159 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcc91be6-98d0-46c5-983b-5ba6b99b74d9 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:38.428467 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca8be285-1902-403a-b50b-e71a3e12db5d admin admin] RBAC: Authorizing `identity:create_grant(project_id=3fa41cc81c4147c287a44d7cefa6e77b, user_id=f91680f344214808954493dc7e276a5f, role_id=a6e562cdc96c4be691d638990f8403fd)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:38.435224 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca8be285-1902-403a-b50b-e71a3e12db5d admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:38.466559 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 956/1912] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:34:38 2022] PUT /identity/v3/projects/3fa41cc81c4147c287a44d7cefa6e77b/users/f91680f344214808954493dc7e276a5f/roles/a6e562cdc96c4be691d638990f8403fd => generated 0 bytes in 356 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:34:38.475380 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca8be285-1902-403a-b50b-e71a3e12db5d admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:38.478523 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-ca8be285-1902-403a-b50b-e71a3e12db5d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:38.498458 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-bcc91be6-98d0-46c5-983b-5ba6b99b74d9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4dcfb6941df346f092934f928e5380c8'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:38.510247 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-bcc91be6-98d0-46c5-983b-5ba6b99b74d9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4dcfb6941df346f092934f928e5380c8'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:38.512785 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=85,keystone:DELETE=14,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:34:38.527953 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-bcc91be6-98d0-46c5-983b-5ba6b99b74d9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4dcfb6941df346f092934f928e5380c8'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:38.529668 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-bcc91be6-98d0-46c5-983b-5ba6b99b74d9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4dcfb6941df346f092934f928e5380c8'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:38.535448 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-bcc91be6-98d0-46c5-983b-5ba6b99b74d9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4dcfb6941df346f092934f928e5380c8'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:38.538732 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:34:38.623555 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 957/1913] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:34:38 2022] DELETE /identity/v3/users/4dcfb6941df346f092934f928e5380c8 => generated 0 bytes in 373 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:34:38.623555 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcc91be6-98d0-46c5-983b-5ba6b99b74d9 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:38.623555 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-bcc91be6-98d0-46c5-983b-5ba6b99b74d9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:38.720428 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca8be285-1902-403a-b50b-e71a3e12db5d admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:38.808858 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd6a7b2a-6f13-4652-814f-7513b7430fb5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:38.809803 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd6a7b2a-6f13-4652-814f-7513b7430fb5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:38.810178 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd6a7b2a-6f13-4652-814f-7513b7430fb5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:38.810650 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd6a7b2a-6f13-4652-814f-7513b7430fb5 admin admin] PATH_INFO: `/v3/projects/550a5a55876e4f4db25b61677df716fd` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:38.812537 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd6a7b2a-6f13-4652-814f-7513b7430fb5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=550a5a55876e4f4db25b61677df716fd)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:38.815475 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd6a7b2a-6f13-4652-814f-7513b7430fb5 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:38.884321 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcc91be6-98d0-46c5-983b-5ba6b99b74d9 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:38.920541 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0297a4aa-1908-446e-8013-4ccba3160fb9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:38.921290 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0297a4aa-1908-446e-8013-4ccba3160fb9 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:38.929742 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0297a4aa-1908-446e-8013-4ccba3160fb9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:38.930373 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0297a4aa-1908-446e-8013-4ccba3160fb9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:38.931639 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0297a4aa-1908-446e-8013-4ccba3160fb9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:38.940064 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0297a4aa-1908-446e-8013-4ccba3160fb9 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:38.955705 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-cd6a7b2a-6f13-4652-814f-7513b7430fb5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '550a5a55876e4f4db25b61677df716fd', 'request_id': 'req-cd6a7b2a-6f13-4652-814f-7513b7430fb5'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:38.979938 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-cd6a7b2a-6f13-4652-814f-7513b7430fb5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '550a5a55876e4f4db25b61677df716fd', 'request_id': 'req-cd6a7b2a-6f13-4652-814f-7513b7430fb5'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:38.981093 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 958/1914] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:34:38 2022] GET /identity/v3/roles => generated 1420 bytes in 361 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:34:38.983040 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0297a4aa-1908-446e-8013-4ccba3160fb9 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:39.001115 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-cd6a7b2a-6f13-4652-814f-7513b7430fb5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '550a5a55876e4f4db25b61677df716fd', 'request_id': 'req-cd6a7b2a-6f13-4652-814f-7513b7430fb5'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:39.022556 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-cd6a7b2a-6f13-4652-814f-7513b7430fb5 admin admin] The token cache is being invalidate because project 550a5a55876e4f4db25b61677df716fd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:34:39.022839 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-cd6a7b2a-6f13-4652-814f-7513b7430fb5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:39.023897 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-cd6a7b2a-6f13-4652-814f-7513b7430fb5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:39.031974 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 957/1915] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:34:38 2022] DELETE /identity/v3/projects/550a5a55876e4f4db25b61677df716fd => generated 0 bytes in 560 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:34:39.032673 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd6a7b2a-6f13-4652-814f-7513b7430fb5 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:39.033821 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0297a4aa-1908-446e-8013-4ccba3160fb9 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:39.039261 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-cd6a7b2a-6f13-4652-814f-7513b7430fb5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:39.147997 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:34:39.226171 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd6a7b2a-6f13-4652-814f-7513b7430fb5 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:39.250326 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07faf588-f597-4233-aed6-8787aee2bee3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:39.251944 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07faf588-f597-4233-aed6-8787aee2bee3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:39.252903 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07faf588-f597-4233-aed6-8787aee2bee3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:39.253364 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07faf588-f597-4233-aed6-8787aee2bee3 admin admin] PATH_INFO: `/v3/users/2100fdb847d34eb796717007b738673e` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:39.259379 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07faf588-f597-4233-aed6-8787aee2bee3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2100fdb847d34eb796717007b738673e)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:39.269458 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07faf588-f597-4233-aed6-8787aee2bee3 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:39.283849 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e2ef288-9653-426e-bd87-8752157b607e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:39.285375 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e2ef288-9653-426e-bd87-8752157b607e admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:39.287267 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e2ef288-9653-426e-bd87-8752157b607e admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:39.288132 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e2ef288-9653-426e-bd87-8752157b607e admin admin] PATH_INFO: `/v3/projects/3fa41cc81c4147c287a44d7cefa6e77b/users/f91680f344214808954493dc7e276a5f/roles/a6e562cdc96c4be691d638990f8403fd` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:39.294373 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e2ef288-9653-426e-bd87-8752157b607e admin admin] RBAC: Authorizing `identity:create_grant(project_id=3fa41cc81c4147c287a44d7cefa6e77b, user_id=f91680f344214808954493dc7e276a5f, role_id=a6e562cdc96c4be691d638990f8403fd)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:39.306843 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e2ef288-9653-426e-bd87-8752157b607e admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:39.333996 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 959/1916] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:34:38 2022] PUT /identity/v3/projects/3fa41cc81c4147c287a44d7cefa6e77b/users/f91680f344214808954493dc7e276a5f/roles/a6e562cdc96c4be691d638990f8403fd => generated 0 bytes in 352 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:34:39.352248 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c85f1a1a-fe46-414c-ab87-7632601810b3 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:39.353446 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c85f1a1a-fe46-414c-ab87-7632601810b3 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:39.354003 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c85f1a1a-fe46-414c-ab87-7632601810b3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:39.438271 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-c85f1a1a-fe46-414c-ab87-7632601810b3 None None] MFA Rules not processed for user `f91680f344214808954493dc7e276a5f`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:34:39.439202 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-07faf588-f597-4233-aed6-8787aee2bee3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2100fdb847d34eb796717007b738673e'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:39.441027 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-c85f1a1a-fe46-414c-ab87-7632601810b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:39.467982 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-07faf588-f597-4233-aed6-8787aee2bee3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2100fdb847d34eb796717007b738673e'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:39.470013 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-07faf588-f597-4233-aed6-8787aee2bee3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2100fdb847d34eb796717007b738673e'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:39.471201 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-07faf588-f597-4233-aed6-8787aee2bee3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2100fdb847d34eb796717007b738673e'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:39.486851 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-07faf588-f597-4233-aed6-8787aee2bee3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2100fdb847d34eb796717007b738673e'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:39.535072 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 958/1917] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:34:39 2022] DELETE /identity/v3/users/2100fdb847d34eb796717007b738673e => generated 0 bytes in 501 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:34:39.566804 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:INSERT=2,keystone:DELETE=3,keystone:SELECT=95 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:34:39.617861 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 960/1918] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:34:39 2022] POST /identity/v3/auth/tokens => generated 3313 bytes in 266 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:34:39.631095 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07faf588-f597-4233-aed6-8787aee2bee3 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:39.632806 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-07faf588-f597-4233-aed6-8787aee2bee3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:39.705952 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c85f1a1a-fe46-414c-ab87-7632601810b3 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:39.710609 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-c85f1a1a-fe46-414c-ab87-7632601810b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:39.788788 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07faf588-f597-4233-aed6-8787aee2bee3 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:39.812169 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e4a9b99-0e9c-4c1d-9cf0-c4e2cd7ae893 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dc226f7ddcb49ecacb51ce57fdfc11b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:39.814363 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e4a9b99-0e9c-4c1d-9cf0-c4e2cd7ae893 service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:39.815416 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e4a9b99-0e9c-4c1d-9cf0-c4e2cd7ae893 service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:39.826056 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e4a9b99-0e9c-4c1d-9cf0-c4e2cd7ae893 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:39.891138 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c85f1a1a-fe46-414c-ab87-7632601810b3 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:40.030795 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd4c43f2-7295-4838-9f82-9f4b349093db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:40.032086 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd4c43f2-7295-4838-9f82-9f4b349093db admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:40.032739 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd4c43f2-7295-4838-9f82-9f4b349093db admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:40.033443 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd4c43f2-7295-4838-9f82-9f4b349093db admin admin] PATH_INFO: `/v3/projects/2300c43f86074c3b9babc975addbc686` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:40.035776 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd4c43f2-7295-4838-9f82-9f4b349093db admin admin] RBAC: Authorizing `identity:delete_project(project_id=2300c43f86074c3b9babc975addbc686)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:40.040756 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd4c43f2-7295-4838-9f82-9f4b349093db admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:40.041579 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e4a9b99-0e9c-4c1d-9cf0-c4e2cd7ae893 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:40.043382 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e4a9b99-0e9c-4c1d-9cf0-c4e2cd7ae893 service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:40.064645 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 959/1919] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:34:39 2022] GET /identity/v3/auth/tokens => generated 3313 bytes in 434 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 21:34:40.146322 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-fd4c43f2-7295-4838-9f82-9f4b349093db admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2300c43f86074c3b9babc975addbc686', 'request_id': 'req-fd4c43f2-7295-4838-9f82-9f4b349093db'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:40.150520 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e4a9b99-0e9c-4c1d-9cf0-c4e2cd7ae893 service cinder] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:40.163866 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-fd4c43f2-7295-4838-9f82-9f4b349093db admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2300c43f86074c3b9babc975addbc686', 'request_id': 'req-fd4c43f2-7295-4838-9f82-9f4b349093db'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:40.165078 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e4a9b99-0e9c-4c1d-9cf0-c4e2cd7ae893 service cinder] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:40.181199 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-fd4c43f2-7295-4838-9f82-9f4b349093db admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2300c43f86074c3b9babc975addbc686', 'request_id': 'req-fd4c43f2-7295-4838-9f82-9f4b349093db'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:40.186046 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:34:40.204507 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-fd4c43f2-7295-4838-9f82-9f4b349093db admin admin] The token cache is being invalidate because project 2300c43f86074c3b9babc975addbc686 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81434) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:34:40.204815 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-fd4c43f2-7295-4838-9f82-9f4b349093db admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:40.205696 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-fd4c43f2-7295-4838-9f82-9f4b349093db admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:40.212772 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 961/1920] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:34:39 2022] DELETE /identity/v3/projects/2300c43f86074c3b9babc975addbc686 => generated 0 bytes in 504 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 21:34:40.215572 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4300ce1-af27-486f-9206-8a8a53b9b76d None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:40.215998 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4300ce1-af27-486f-9206-8a8a53b9b76d None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:40.216384 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4300ce1-af27-486f-9206-8a8a53b9b76d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:40.246172 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-a4300ce1-af27-486f-9206-8a8a53b9b76d None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:34:40.247687 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-a4300ce1-af27-486f-9206-8a8a53b9b76d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:40.281217 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab168ce1-ac5d-4ad6-80a6-ad53aa5dfc7d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dc226f7ddcb49ecacb51ce57fdfc11b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:40.281732 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab168ce1-ac5d-4ad6-80a6-ad53aa5dfc7d service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:40.282159 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab168ce1-ac5d-4ad6-80a6-ad53aa5dfc7d service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:40.282428 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab168ce1-ac5d-4ad6-80a6-ad53aa5dfc7d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:40.284978 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-ab168ce1-ac5d-4ad6-80a6-ad53aa5dfc7d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:40.289272 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:34:40.397902 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 962/1921] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:34:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 21:34:40.400363 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b244bfa2-608e-42a8-b6e9-ba7952bfa0b6 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:40.400898 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b244bfa2-608e-42a8-b6e9-ba7952bfa0b6 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:40.401413 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b244bfa2-608e-42a8-b6e9-ba7952bfa0b6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:40.441613 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-b244bfa2-608e-42a8-b6e9-ba7952bfa0b6 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:34:40.443553 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-b244bfa2-608e-42a8-b6e9-ba7952bfa0b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:40.461870 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab168ce1-ac5d-4ad6-80a6-ad53aa5dfc7d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:40.462215 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 963/1922] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:34:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 64 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:34:40.464069 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab168ce1-ac5d-4ad6-80a6-ad53aa5dfc7d service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:40.468090 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b244bfa2-608e-42a8-b6e9-ba7952bfa0b6 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:40.482042 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b244bfa2-608e-42a8-b6e9-ba7952bfa0b6 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:40.488928 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d43d4ef-c345-499b-a3f8-60bb652c2a14 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:40.491367 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d43d4ef-c345-499b-a3f8-60bb652c2a14 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:40.491560 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 960/1923] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:34:40 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 341 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 21:34:40.492009 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d43d4ef-c345-499b-a3f8-60bb652c2a14 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:40.492398 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d43d4ef-c345-499b-a3f8-60bb652c2a14 admin admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:40.493168 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d43d4ef-c345-499b-a3f8-60bb652c2a14 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:34:40.493546 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d43d4ef-c345-499b-a3f8-60bb652c2a14 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:40.496566 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d43d4ef-c345-499b-a3f8-60bb652c2a14 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:40.522551 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 964/1924] 10.222.0.90 () {58 vars in 1153 bytes} [Mon Aug 29 21:34:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:34:40.524923 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab168ce1-ac5d-4ad6-80a6-ad53aa5dfc7d service cinder] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:40.560752 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab168ce1-ac5d-4ad6-80a6-ad53aa5dfc7d service cinder] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:40.573295 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daa84a1a-7caa-4243-9e9c-1e927a95f1ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:40.574476 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa84a1a-7caa-4243-9e9c-1e927a95f1ff admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:40.574929 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa84a1a-7caa-4243-9e9c-1e927a95f1ff admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:40.575211 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa84a1a-7caa-4243-9e9c-1e927a95f1ff admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:40.576094 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daa84a1a-7caa-4243-9e9c-1e927a95f1ff admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:40.578256 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daa84a1a-7caa-4243-9e9c-1e927a95f1ff admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:40.639983 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 961/1925] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:34:40 2022] POST /identity/v3/projects => generated 387 bytes in 116 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:34:40.645979 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d43d4ef-c345-499b-a3f8-60bb652c2a14 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:40.658811 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d43d4ef-c345-499b-a3f8-60bb652c2a14 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:40.764101 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daa84a1a-7caa-4243-9e9c-1e927a95f1ff admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:40.768168 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-daa84a1a-7caa-4243-9e9c-1e927a95f1ff admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:40.833685 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecf78808-b22f-4bf4-8207-73c2886fbd5f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:40.834585 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf78808-b22f-4bf4-8207-73c2886fbd5f admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:40.834732 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf78808-b22f-4bf4-8207-73c2886fbd5f admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:40.838158 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf78808-b22f-4bf4-8207-73c2886fbd5f admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:40.839062 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecf78808-b22f-4bf4-8207-73c2886fbd5f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:40.847870 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecf78808-b22f-4bf4-8207-73c2886fbd5f admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:40.929884 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 965/1926] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:34:40 2022] POST /identity/v3/users => generated 294 bytes in 281 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:34:40.931939 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecf78808-b22f-4bf4-8207-73c2886fbd5f admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:40.977047 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecf78808-b22f-4bf4-8207-73c2886fbd5f admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:40.985829 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3176752b-a7ee-401d-82f9-ceaeefe74e82 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:40.987972 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3176752b-a7ee-401d-82f9-ceaeefe74e82 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:40.988499 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3176752b-a7ee-401d-82f9-ceaeefe74e82 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:40.988976 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3176752b-a7ee-401d-82f9-ceaeefe74e82 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:40.991304 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3176752b-a7ee-401d-82f9-ceaeefe74e82 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:40.993428 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daa84a1a-7caa-4243-9e9c-1e927a95f1ff admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:41.003420 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3176752b-a7ee-401d-82f9-ceaeefe74e82 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:41.008915 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9319681d-4843-4a70-bae8-830260f96d13 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:41.017691 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9319681d-4843-4a70-bae8-830260f96d13 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:41.017955 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9319681d-4843-4a70-bae8-830260f96d13 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:41.018246 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9319681d-4843-4a70-bae8-830260f96d13 admin admin] PATH_INFO: `/v3/users/31bb2e8689d5408b924b2cb7aa98da1c` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:41.019804 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9319681d-4843-4a70-bae8-830260f96d13 admin admin] RBAC: Authorizing `identity:delete_user(user_id=31bb2e8689d5408b924b2cb7aa98da1c)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:41.021874 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9319681d-4843-4a70-bae8-830260f96d13 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:41.058444 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 966/1927] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:34:40 2022] GET /identity/v3/roles => generated 1420 bytes in 127 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:34:41.066979 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3176752b-a7ee-401d-82f9-ceaeefe74e82 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:41.137639 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3176752b-a7ee-401d-82f9-ceaeefe74e82 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:41.170986 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c5dfbe3-8047-4f74-a9b0-f821eafe0020 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:41.172038 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c5dfbe3-8047-4f74-a9b0-f821eafe0020 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:41.172543 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c5dfbe3-8047-4f74-a9b0-f821eafe0020 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:41.173027 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c5dfbe3-8047-4f74-a9b0-f821eafe0020 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:41.177705 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c5dfbe3-8047-4f74-a9b0-f821eafe0020 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:41.184398 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c5dfbe3-8047-4f74-a9b0-f821eafe0020 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:41.259116 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 967/1928] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:34:41 2022] GET /identity/v3/roles => generated 1420 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:34:41.283240 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-9319681d-4843-4a70-bae8-830260f96d13 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '31bb2e8689d5408b924b2cb7aa98da1c'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:41.288571 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c5dfbe3-8047-4f74-a9b0-f821eafe0020 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:41.295399 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=92,keystone:INSERT=1,keystone:DELETE=7 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:34:41.319366 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c5dfbe3-8047-4f74-a9b0-f821eafe0020 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:41.325074 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-9319681d-4843-4a70-bae8-830260f96d13 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '31bb2e8689d5408b924b2cb7aa98da1c'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:41.326463 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-9319681d-4843-4a70-bae8-830260f96d13 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '31bb2e8689d5408b924b2cb7aa98da1c'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:41.328044 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-9319681d-4843-4a70-bae8-830260f96d13 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '31bb2e8689d5408b924b2cb7aa98da1c'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:41.338677 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a12abf9c-32d1-481d-bb76-0e8f81e6c776 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:41.339853 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a12abf9c-32d1-481d-bb76-0e8f81e6c776 admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:41.340109 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a12abf9c-32d1-481d-bb76-0e8f81e6c776 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:41.340498 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a12abf9c-32d1-481d-bb76-0e8f81e6c776 admin admin] PATH_INFO: `/v3/projects/5b020c39ae3f4d1fba39663bc5d05978/users/c0b75929a53b42b09cec886587a5888d/roles/02e242d5520740d4b77a5f4008c7eb57` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:41.347253 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-9319681d-4843-4a70-bae8-830260f96d13 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '31bb2e8689d5408b924b2cb7aa98da1c'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:41.381753 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 962/1929] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:34:40 2022] DELETE /identity/v3/users/31bb2e8689d5408b924b2cb7aa98da1c => generated 0 bytes in 618 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 21:34:41.387157 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a12abf9c-32d1-481d-bb76-0e8f81e6c776 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5b020c39ae3f4d1fba39663bc5d05978, user_id=c0b75929a53b42b09cec886587a5888d, role_id=02e242d5520740d4b77a5f4008c7eb57)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:41.388977 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a47093b-977d-4547-a533-9ad816d5b252 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:41.389324 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a47093b-977d-4547-a533-9ad816d5b252 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:41.389769 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a47093b-977d-4547-a533-9ad816d5b252 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:41.392587 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a12abf9c-32d1-481d-bb76-0e8f81e6c776 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:41.429521 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-8a47093b-977d-4547-a533-9ad816d5b252 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:34:41.433042 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-8a47093b-977d-4547-a533-9ad816d5b252 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:41.443844 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 968/1930] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:34:41 2022] PUT /identity/v3/projects/5b020c39ae3f4d1fba39663bc5d05978/users/c0b75929a53b42b09cec886587a5888d/roles/02e242d5520740d4b77a5f4008c7eb57 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 21:34:41.458476 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a12abf9c-32d1-481d-bb76-0e8f81e6c776 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:41.471690 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-a12abf9c-32d1-481d-bb76-0e8f81e6c776 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:41.549977 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 963/1931] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:34:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:34:41.553889 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2539b5d7-1e2d-4182-84b5-614819ffac27 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:41.554551 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2539b5d7-1e2d-4182-84b5-614819ffac27 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:41.555186 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2539b5d7-1e2d-4182-84b5-614819ffac27 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:41.616358 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:34:41.646458 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a12abf9c-32d1-481d-bb76-0e8f81e6c776 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:41.650443 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-2539b5d7-1e2d-4182-84b5-614819ffac27 None None] MFA Rules not processed for user `c0b75929a53b42b09cec886587a5888d`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:34:41.653614 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-2539b5d7-1e2d-4182-84b5-614819ffac27 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:41.654417 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52aba77e-50d6-4dfe-b089-42e0ce6a7a56 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:41.655528 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52aba77e-50d6-4dfe-b089-42e0ce6a7a56 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:41.655983 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52aba77e-50d6-4dfe-b089-42e0ce6a7a56 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:41.656503 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52aba77e-50d6-4dfe-b089-42e0ce6a7a56 admin admin] PATH_INFO: `/v3/projects/7d67ea5dfcac44cab179846e256a44e4` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:41.658759 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52aba77e-50d6-4dfe-b089-42e0ce6a7a56 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7d67ea5dfcac44cab179846e256a44e4)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:41.660720 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52aba77e-50d6-4dfe-b089-42e0ce6a7a56 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:41.755591 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-52aba77e-50d6-4dfe-b089-42e0ce6a7a56 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7d67ea5dfcac44cab179846e256a44e4', 'request_id': 'req-52aba77e-50d6-4dfe-b089-42e0ce6a7a56'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:41.768957 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-52aba77e-50d6-4dfe-b089-42e0ce6a7a56 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7d67ea5dfcac44cab179846e256a44e4', 'request_id': 'req-52aba77e-50d6-4dfe-b089-42e0ce6a7a56'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:41.776188 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-52aba77e-50d6-4dfe-b089-42e0ce6a7a56 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7d67ea5dfcac44cab179846e256a44e4', 'request_id': 'req-52aba77e-50d6-4dfe-b089-42e0ce6a7a56'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:41.795769 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-52aba77e-50d6-4dfe-b089-42e0ce6a7a56 admin admin] The token cache is being invalidate because project 7d67ea5dfcac44cab179846e256a44e4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81434) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:34:41.796132 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-52aba77e-50d6-4dfe-b089-42e0ce6a7a56 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:41.796828 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-52aba77e-50d6-4dfe-b089-42e0ce6a7a56 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:41.799563 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 969/1932] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:34:41 2022] DELETE /identity/v3/projects/7d67ea5dfcac44cab179846e256a44e4 => generated 0 bytes in 356 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:34:41.800388 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52aba77e-50d6-4dfe-b089-42e0ce6a7a56 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:41.802536 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-52aba77e-50d6-4dfe-b089-42e0ce6a7a56 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:41.820562 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 964/1933] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:34:41 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 266 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:34:41.820562 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2089e6f7-a691-4b78-a7a7-f080e299eb5d None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:41.820562 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2089e6f7-a691-4b78-a7a7-f080e299eb5d None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:41.820934 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2089e6f7-a691-4b78-a7a7-f080e299eb5d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:41.862460 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-2089e6f7-a691-4b78-a7a7-f080e299eb5d None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:34:41.865116 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-2089e6f7-a691-4b78-a7a7-f080e299eb5d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:41.943611 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52aba77e-50d6-4dfe-b089-42e0ce6a7a56 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:41.948743 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78d07629-96a7-4b85-b9e1-7a3f5430a4ad service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:41.949444 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78d07629-96a7-4b85-b9e1-7a3f5430a4ad service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:41.949729 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78d07629-96a7-4b85-b9e1-7a3f5430a4ad service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:41.950059 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78d07629-96a7-4b85-b9e1-7a3f5430a4ad service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:41.952789 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-78d07629-96a7-4b85-b9e1-7a3f5430a4ad service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:42.044396 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:DELETE=3,keystone:SELECT=96,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:34:42.072231 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 965/1934] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:34:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 247 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:34:42.072231 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2089e6f7-a691-4b78-a7a7-f080e299eb5d None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:42.072231 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-2089e6f7-a691-4b78-a7a7-f080e299eb5d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:42.090141 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2089e6f7-a691-4b78-a7a7-f080e299eb5d None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:42.105107 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0ab7c32-8e63-4785-9af7-578a2a823443 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:42.106244 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ab7c32-8e63-4785-9af7-578a2a823443 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:42.107091 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ab7c32-8e63-4785-9af7-578a2a823443 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:42.107591 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ab7c32-8e63-4785-9af7-578a2a823443 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:42.108626 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0ab7c32-8e63-4785-9af7-578a2a823443 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:42.110832 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0ab7c32-8e63-4785-9af7-578a2a823443 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:42.131554 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78d07629-96a7-4b85-b9e1-7a3f5430a4ad service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:42.134206 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78d07629-96a7-4b85-b9e1-7a3f5430a4ad service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:42.155018 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 966/1935] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:34:42 2022] POST /identity/v3/projects => generated 387 bytes in 89 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:34:42.156112 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc2ad07-1572-4280-9373-b39d57c5d9d3 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:42.156399 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc2ad07-1572-4280-9373-b39d57c5d9d3 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:42.156677 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc2ad07-1572-4280-9373-b39d57c5d9d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:42.165680 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 970/1936] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:34:41 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 366 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:34:42.167251 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78d07629-96a7-4b85-b9e1-7a3f5430a4ad service neutron] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:42.186663 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-2dc2ad07-1572-4280-9373-b39d57c5d9d3 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:34:42.188530 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-2dc2ad07-1572-4280-9373-b39d57c5d9d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:42.189104 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78d07629-96a7-4b85-b9e1-7a3f5430a4ad service neutron] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:42.268380 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:34:42.402569 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 967/1937] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:34:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 247 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:34:42.409037 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dc2ad07-1572-4280-9373-b39d57c5d9d3 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:42.422421 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7179a84a-8b1b-4e2f-b7fe-20421a06ab9f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:42.425035 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7179a84a-8b1b-4e2f-b7fe-20421a06ab9f admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:42.425624 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7179a84a-8b1b-4e2f-b7fe-20421a06ab9f admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:42.426221 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7179a84a-8b1b-4e2f-b7fe-20421a06ab9f admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:42.428052 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7179a84a-8b1b-4e2f-b7fe-20421a06ab9f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:42.432113 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7179a84a-8b1b-4e2f-b7fe-20421a06ab9f admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:42.440337 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dc2ad07-1572-4280-9373-b39d57c5d9d3 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:42.459591 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cd75da1-da2c-4437-bcc2-742856680edd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:42.460284 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cd75da1-da2c-4437-bcc2-742856680edd admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:42.460577 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cd75da1-da2c-4437-bcc2-742856680edd admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:42.460930 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cd75da1-da2c-4437-bcc2-742856680edd admin admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:42.461599 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cd75da1-da2c-4437-bcc2-742856680edd admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:34:42.461975 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cd75da1-da2c-4437-bcc2-742856680edd admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:42.465995 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cd75da1-da2c-4437-bcc2-742856680edd admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:42.508754 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 971/1938] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:34:42 2022] POST /identity/v3/users => generated 300 bytes in 342 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:34:42.514750 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7179a84a-8b1b-4e2f-b7fe-20421a06ab9f admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:42.522954 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 968/1939] 10.222.0.90 () {58 vars in 1153 bytes} [Mon Aug 29 21:34:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 114 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:34:42.530268 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cd75da1-da2c-4437-bcc2-742856680edd admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:42.534785 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7179a84a-8b1b-4e2f-b7fe-20421a06ab9f admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:42.548349 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cd75da1-da2c-4437-bcc2-742856680edd admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:42.549313 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4adfa137-8f7b-4380-b699-695ee4389254 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:42.552308 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4adfa137-8f7b-4380-b699-695ee4389254 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:42.554912 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4adfa137-8f7b-4380-b699-695ee4389254 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:42.555266 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4adfa137-8f7b-4380-b699-695ee4389254 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:42.557448 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4adfa137-8f7b-4380-b699-695ee4389254 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:42.562030 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4adfa137-8f7b-4380-b699-695ee4389254 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:42.564300 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dacd5dad-c889-450d-9b52-22f8dba3258b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:42.565752 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dacd5dad-c889-450d-9b52-22f8dba3258b admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:42.565903 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dacd5dad-c889-450d-9b52-22f8dba3258b admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:42.566344 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dacd5dad-c889-450d-9b52-22f8dba3258b admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:42.568330 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dacd5dad-c889-450d-9b52-22f8dba3258b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:42.572426 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dacd5dad-c889-450d-9b52-22f8dba3258b admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:42.592246 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 972/1940] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:34:42 2022] GET /identity/v3/roles => generated 1420 bytes in 77 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:34:42.593698 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4adfa137-8f7b-4380-b699-695ee4389254 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:42.649262 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4adfa137-8f7b-4380-b699-695ee4389254 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:42.665064 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 969/1941] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:34:42 2022] POST /identity/v3/projects => generated 381 bytes in 136 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 21:34:42.668064 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dacd5dad-c889-450d-9b52-22f8dba3258b admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:42.679835 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4738debb-df3f-4ede-af1a-dfe204890a67 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:42.681093 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4738debb-df3f-4ede-af1a-dfe204890a67 admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:42.681581 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4738debb-df3f-4ede-af1a-dfe204890a67 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:42.682507 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4738debb-df3f-4ede-af1a-dfe204890a67 admin admin] PATH_INFO: `/v3/projects/a485a9d3f22046acb8cf932f589de51d/users/d115ddd16b73493691b2a697bd4fac84/roles/a6e562cdc96c4be691d638990f8403fd` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:42.772271 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4738debb-df3f-4ede-af1a-dfe204890a67 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a485a9d3f22046acb8cf932f589de51d, user_id=d115ddd16b73493691b2a697bd4fac84, role_id=a6e562cdc96c4be691d638990f8403fd)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:42.781008 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4738debb-df3f-4ede-af1a-dfe204890a67 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:42.838876 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 973/1942] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:34:42 2022] PUT /identity/v3/projects/a485a9d3f22046acb8cf932f589de51d/users/d115ddd16b73493691b2a697bd4fac84/roles/a6e562cdc96c4be691d638990f8403fd => generated 0 bytes in 246 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:34:42.844495 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4738debb-df3f-4ede-af1a-dfe204890a67 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:42.945820 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dacd5dad-c889-450d-9b52-22f8dba3258b admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:43.033364 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4738debb-df3f-4ede-af1a-dfe204890a67 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:43.048931 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4637f101-c364-40f9-a85f-be843fd681ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:43.053737 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4637f101-c364-40f9-a85f-be843fd681ab admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:43.054286 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4637f101-c364-40f9-a85f-be843fd681ab admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:43.055898 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4637f101-c364-40f9-a85f-be843fd681ab admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:43.057073 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4637f101-c364-40f9-a85f-be843fd681ab admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:43.065939 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4637f101-c364-40f9-a85f-be843fd681ab admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:43.103955 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:34:43.109065 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 974/1943] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:34:42 2022] GET /identity/v3/roles => generated 1420 bytes in 265 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 29 21:34:43.113647 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4637f101-c364-40f9-a85f-be843fd681ab admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:43.129460 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4f6418f-16eb-48b1-9a0a-53389229a5d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:43.130855 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4f6418f-16eb-48b1-9a0a-53389229a5d4 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:43.131172 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4f6418f-16eb-48b1-9a0a-53389229a5d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:43.131702 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4f6418f-16eb-48b1-9a0a-53389229a5d4 admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:43.132365 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4f6418f-16eb-48b1-9a0a-53389229a5d4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:43.134486 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4f6418f-16eb-48b1-9a0a-53389229a5d4 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:43.144122 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4637f101-c364-40f9-a85f-be843fd681ab admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:43.158376 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-719e031c-0f52-47e2-9a0b-0718c3ec76c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:43.159533 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-719e031c-0f52-47e2-9a0b-0718c3ec76c1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:43.160106 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-719e031c-0f52-47e2-9a0b-0718c3ec76c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:43.160656 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-719e031c-0f52-47e2-9a0b-0718c3ec76c1 admin admin] PATH_INFO: `/v3/projects/a485a9d3f22046acb8cf932f589de51d/users/d115ddd16b73493691b2a697bd4fac84/roles/a6e562cdc96c4be691d638990f8403fd` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:43.164845 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-719e031c-0f52-47e2-9a0b-0718c3ec76c1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a485a9d3f22046acb8cf932f589de51d, user_id=d115ddd16b73493691b2a697bd4fac84, role_id=a6e562cdc96c4be691d638990f8403fd)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:43.170918 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-719e031c-0f52-47e2-9a0b-0718c3ec76c1 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:43.176917 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 970/1944] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:34:42 2022] POST /identity/v3/users => generated 291 bytes in 510 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:34:43.186804 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4f6418f-16eb-48b1-9a0a-53389229a5d4 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:43.195475 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 975/1945] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:34:43 2022] PUT /identity/v3/projects/a485a9d3f22046acb8cf932f589de51d/users/d115ddd16b73493691b2a697bd4fac84/roles/a6e562cdc96c4be691d638990f8403fd => generated 0 bytes in 82 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:34:43.211733 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4f6418f-16eb-48b1-9a0a-53389229a5d4 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:43.216087 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2af8f81-1af3-4f72-9abf-ec014ec5b217 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:43.216414 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2af8f81-1af3-4f72-9abf-ec014ec5b217 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:43.216832 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2af8f81-1af3-4f72-9abf-ec014ec5b217 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:43.303452 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e049893f-2b8a-4336-915f-7f98ccbe8e25 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:43.304139 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e049893f-2b8a-4336-915f-7f98ccbe8e25 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:43.304423 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e049893f-2b8a-4336-915f-7f98ccbe8e25 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:43.304708 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e049893f-2b8a-4336-915f-7f98ccbe8e25 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:43.305391 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e049893f-2b8a-4336-915f-7f98ccbe8e25 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:43.307069 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e049893f-2b8a-4336-915f-7f98ccbe8e25 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:43.320695 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 971/1946] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:34:43 2022] GET /identity/v3/roles => generated 1420 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:34:43.326707 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e049893f-2b8a-4336-915f-7f98ccbe8e25 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:43.354317 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e049893f-2b8a-4336-915f-7f98ccbe8e25 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:43.367077 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea8ab88b-312c-4ded-a0df-952cf9953f89 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:43.367810 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea8ab88b-312c-4ded-a0df-952cf9953f89 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:43.368182 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea8ab88b-312c-4ded-a0df-952cf9953f89 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:43.368471 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea8ab88b-312c-4ded-a0df-952cf9953f89 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:43.369398 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea8ab88b-312c-4ded-a0df-952cf9953f89 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:43.372184 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea8ab88b-312c-4ded-a0df-952cf9953f89 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:43.385847 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-e2af8f81-1af3-4f72-9abf-ec014ec5b217 None None] MFA Rules not processed for user `d115ddd16b73493691b2a697bd4fac84`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:34:43.394777 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 972/1947] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:34:43 2022] GET /identity/v3/roles => generated 1420 bytes in 65 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 29 21:34:43.394942 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-e2af8f81-1af3-4f72-9abf-ec014ec5b217 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:43.415015 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea8ab88b-312c-4ded-a0df-952cf9953f89 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:43.435040 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea8ab88b-312c-4ded-a0df-952cf9953f89 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:43.465457 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d97a541-705c-4387-8eda-86b1e4f15b8e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:43.466625 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d97a541-705c-4387-8eda-86b1e4f15b8e admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:43.467091 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d97a541-705c-4387-8eda-86b1e4f15b8e admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:43.467558 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d97a541-705c-4387-8eda-86b1e4f15b8e admin admin] PATH_INFO: `/v3/projects/01ad67f60b6c459a979014fbeaf203e1/users/ba6cbf68ca7245cdaeb1cc6975b23d0d/roles/02e242d5520740d4b77a5f4008c7eb57` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:43.528697 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d97a541-705c-4387-8eda-86b1e4f15b8e admin admin] RBAC: Authorizing `identity:create_grant(project_id=01ad67f60b6c459a979014fbeaf203e1, user_id=ba6cbf68ca7245cdaeb1cc6975b23d0d, role_id=02e242d5520740d4b77a5f4008c7eb57)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:43.555500 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d97a541-705c-4387-8eda-86b1e4f15b8e admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:43.611860 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 973/1948] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:34:43 2022] PUT /identity/v3/projects/01ad67f60b6c459a979014fbeaf203e1/users/ba6cbf68ca7245cdaeb1cc6975b23d0d/roles/02e242d5520740d4b77a5f4008c7eb57 => generated 0 bytes in 203 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:34:43.632165 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56588f8d-c880-470b-a503-0c82cef4dc3e None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:43.632745 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56588f8d-c880-470b-a503-0c82cef4dc3e None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:43.633297 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56588f8d-c880-470b-a503-0c82cef4dc3e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:43.690242 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:34:43.698324 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-56588f8d-c880-470b-a503-0c82cef4dc3e None None] MFA Rules not processed for user `ba6cbf68ca7245cdaeb1cc6975b23d0d`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:34:43.700419 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-56588f8d-c880-470b-a503-0c82cef4dc3e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:43.723149 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 976/1949] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:34:43 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 508 msecs (HTTP/1.1 201) 6 headers in 385 bytes (3 switches on core 0) Aug 29 21:34:43.747232 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2af8f81-1af3-4f72-9abf-ec014ec5b217 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:43.749844 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-e2af8f81-1af3-4f72-9abf-ec014ec5b217 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:43.904017 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2af8f81-1af3-4f72-9abf-ec014ec5b217 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:43.904850 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 974/1950] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:34:43 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 275 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:34:43.906226 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56588f8d-c880-470b-a503-0c82cef4dc3e None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:43.908529 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-56588f8d-c880-470b-a503-0c82cef4dc3e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:43.991542 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:34:44.003289 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6782ad4e-68fa-47b2-8259-0764e4ddd590 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dc226f7ddcb49ecacb51ce57fdfc11b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:44.005645 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6782ad4e-68fa-47b2-8259-0764e4ddd590 service cinder] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:44.006096 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6782ad4e-68fa-47b2-8259-0764e4ddd590 service cinder] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:44.006491 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6782ad4e-68fa-47b2-8259-0764e4ddd590 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:44.048070 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56588f8d-c880-470b-a503-0c82cef4dc3e None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:44.055722 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb5ee2b7-a664-4a77-b3e8-64dc8c3c615c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:44.056371 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb5ee2b7-a664-4a77-b3e8-64dc8c3c615c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:44.056649 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb5ee2b7-a664-4a77-b3e8-64dc8c3c615c admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:44.056922 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb5ee2b7-a664-4a77-b3e8-64dc8c3c615c admin admin] PATH_INFO: `/v3/projects/5ee76498fc4747369d7342e932f04c39` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:44.058198 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb5ee2b7-a664-4a77-b3e8-64dc8c3c615c admin admin] RBAC: Authorizing `identity:delete_project(project_id=5ee76498fc4747369d7342e932f04c39)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:44.060111 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb5ee2b7-a664-4a77-b3e8-64dc8c3c615c admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:44.119892 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6782ad4e-68fa-47b2-8259-0764e4ddd590 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:44.123385 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6782ad4e-68fa-47b2-8259-0764e4ddd590 service cinder] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:44.141103 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 977/1951] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:34:43 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 397 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:34:44.142531 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6782ad4e-68fa-47b2-8259-0764e4ddd590 service cinder] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:44.149299 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-cb5ee2b7-a664-4a77-b3e8-64dc8c3c615c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5ee76498fc4747369d7342e932f04c39', 'request_id': 'req-cb5ee2b7-a664-4a77-b3e8-64dc8c3c615c'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:44.160490 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6782ad4e-68fa-47b2-8259-0764e4ddd590 service cinder] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:44.165383 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-cb5ee2b7-a664-4a77-b3e8-64dc8c3c615c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5ee76498fc4747369d7342e932f04c39', 'request_id': 'req-cb5ee2b7-a664-4a77-b3e8-64dc8c3c615c'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:44.180903 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-cb5ee2b7-a664-4a77-b3e8-64dc8c3c615c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5ee76498fc4747369d7342e932f04c39', 'request_id': 'req-cb5ee2b7-a664-4a77-b3e8-64dc8c3c615c'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:44.222609 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-cb5ee2b7-a664-4a77-b3e8-64dc8c3c615c admin admin] The token cache is being invalidate because project 5ee76498fc4747369d7342e932f04c39 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:34:44.223106 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-cb5ee2b7-a664-4a77-b3e8-64dc8c3c615c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:44.226168 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-cb5ee2b7-a664-4a77-b3e8-64dc8c3c615c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:44.231308 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 975/1952] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:34:43 2022] DELETE /identity/v3/projects/5ee76498fc4747369d7342e932f04c39 => generated 0 bytes in 325 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:34:44.237289 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb5ee2b7-a664-4a77-b3e8-64dc8c3c615c admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:44.243964 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-cb5ee2b7-a664-4a77-b3e8-64dc8c3c615c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:44.271939 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f0ed3eb-d081-4e51-a0c0-efbb3952b3af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:44.273138 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f0ed3eb-d081-4e51-a0c0-efbb3952b3af admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:44.273745 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f0ed3eb-d081-4e51-a0c0-efbb3952b3af admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:44.274766 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f0ed3eb-d081-4e51-a0c0-efbb3952b3af admin admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:44.276715 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f0ed3eb-d081-4e51-a0c0-efbb3952b3af admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:44.281600 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f0ed3eb-d081-4e51-a0c0-efbb3952b3af admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:44.302539 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:34:44.363721 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 978/1953] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:34:44 2022] POST /identity/v3/projects => generated 381 bytes in 222 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:34:44.368124 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f0ed3eb-d081-4e51-a0c0-efbb3952b3af admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:44.373582 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-7f0ed3eb-d081-4e51-a0c0-efbb3952b3af admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:44.376803 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb5ee2b7-a664-4a77-b3e8-64dc8c3c615c admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:44.500469 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e09659de-e9da-4587-a7a5-79a0120fd483 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:44.501285 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e09659de-e9da-4587-a7a5-79a0120fd483 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:44.501860 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e09659de-e9da-4587-a7a5-79a0120fd483 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:44.502346 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e09659de-e9da-4587-a7a5-79a0120fd483 admin admin] PATH_INFO: `/v3/users/7af84e4542ab46978562428f29c8d933` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:44.504404 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e09659de-e9da-4587-a7a5-79a0120fd483 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7af84e4542ab46978562428f29c8d933)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:44.509836 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e09659de-e9da-4587-a7a5-79a0120fd483 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:44.540299 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:34:44.541589 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f0ed3eb-d081-4e51-a0c0-efbb3952b3af admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:44.555469 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22720e32-a6fb-4d3e-9aaf-f334b4db05c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:44.556215 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22720e32-a6fb-4d3e-9aaf-f334b4db05c7 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:44.556774 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22720e32-a6fb-4d3e-9aaf-f334b4db05c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:44.556998 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22720e32-a6fb-4d3e-9aaf-f334b4db05c7 admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:44.557774 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22720e32-a6fb-4d3e-9aaf-f334b4db05c7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:44.560264 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22720e32-a6fb-4d3e-9aaf-f334b4db05c7 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:44.641732 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-e09659de-e9da-4587-a7a5-79a0120fd483 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7af84e4542ab46978562428f29c8d933'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:44.642492 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 979/1954] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:34:44 2022] POST /identity/v3/users => generated 297 bytes in 275 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:34:44.643621 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22720e32-a6fb-4d3e-9aaf-f334b4db05c7 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:44.645489 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-22720e32-a6fb-4d3e-9aaf-f334b4db05c7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:44.659137 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22720e32-a6fb-4d3e-9aaf-f334b4db05c7 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:44.668397 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b676225-60d5-48f6-aa24-e0a5afe0c41a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:44.669468 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b676225-60d5-48f6-aa24-e0a5afe0c41a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:44.671244 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b676225-60d5-48f6-aa24-e0a5afe0c41a admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:44.671622 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b676225-60d5-48f6-aa24-e0a5afe0c41a admin admin] PATH_INFO: `/v3/users/c0b75929a53b42b09cec886587a5888d` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:44.675666 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b676225-60d5-48f6-aa24-e0a5afe0c41a admin admin] RBAC: Authorizing `identity:delete_user(user_id=c0b75929a53b42b09cec886587a5888d)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:44.680115 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b676225-60d5-48f6-aa24-e0a5afe0c41a admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:44.680820 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-e09659de-e9da-4587-a7a5-79a0120fd483 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7af84e4542ab46978562428f29c8d933'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:44.683486 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-e09659de-e9da-4587-a7a5-79a0120fd483 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7af84e4542ab46978562428f29c8d933'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:44.685168 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-e09659de-e9da-4587-a7a5-79a0120fd483 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7af84e4542ab46978562428f29c8d933'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:44.718173 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-e09659de-e9da-4587-a7a5-79a0120fd483 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7af84e4542ab46978562428f29c8d933'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:44.753141 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 976/1955] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:34:44 2022] DELETE /identity/v3/users/7af84e4542ab46978562428f29c8d933 => generated 0 bytes in 517 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 21:34:44.757978 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e09659de-e9da-4587-a7a5-79a0120fd483 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:44.762034 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-e09659de-e9da-4587-a7a5-79a0120fd483 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:44.812888 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-2b676225-60d5-48f6-aa24-e0a5afe0c41a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c0b75929a53b42b09cec886587a5888d'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:44.827617 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-2b676225-60d5-48f6-aa24-e0a5afe0c41a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c0b75929a53b42b09cec886587a5888d'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:44.850753 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-2b676225-60d5-48f6-aa24-e0a5afe0c41a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c0b75929a53b42b09cec886587a5888d'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:44.851879 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-2b676225-60d5-48f6-aa24-e0a5afe0c41a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c0b75929a53b42b09cec886587a5888d'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:44.852091 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-2b676225-60d5-48f6-aa24-e0a5afe0c41a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c0b75929a53b42b09cec886587a5888d'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:44.897208 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 980/1956] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:34:44 2022] DELETE /identity/v3/users/c0b75929a53b42b09cec886587a5888d => generated 0 bytes in 254 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:34:44.902469 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d27e85a-e23b-4398-bb6e-036470ecd5e9 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:44.902976 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d27e85a-e23b-4398-bb6e-036470ecd5e9 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:44.903256 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d27e85a-e23b-4398-bb6e-036470ecd5e9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:44.951642 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-6d27e85a-e23b-4398-bb6e-036470ecd5e9 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:34:44.952880 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-6d27e85a-e23b-4398-bb6e-036470ecd5e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:45.092585 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=85,keystone:INSERT=5,keystone:DELETE=10 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:34:45.094282 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e09659de-e9da-4587-a7a5-79a0120fd483 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:45.133863 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 981/1957] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:34:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 232 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 21:34:45.173892 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d27e85a-e23b-4398-bb6e-036470ecd5e9 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:45.188600 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-6d27e85a-e23b-4398-bb6e-036470ecd5e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:45.257246 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3167258-a08d-45be-9327-c8292ed63953 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:45.258171 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3167258-a08d-45be-9327-c8292ed63953 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:45.258607 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3167258-a08d-45be-9327-c8292ed63953 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:45.265398 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3167258-a08d-45be-9327-c8292ed63953 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:45.268068 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3167258-a08d-45be-9327-c8292ed63953 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:45.273845 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3167258-a08d-45be-9327-c8292ed63953 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:45.280060 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d27e85a-e23b-4398-bb6e-036470ecd5e9 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:45.284791 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-327f8fc5-eabd-4fd6-a784-577b1b546897 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:45.285173 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-327f8fc5-eabd-4fd6-a784-577b1b546897 service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:45.285451 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-327f8fc5-eabd-4fd6-a784-577b1b546897 service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:45.285722 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-327f8fc5-eabd-4fd6-a784-577b1b546897 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:45.297054 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-327f8fc5-eabd-4fd6-a784-577b1b546897 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:45.297327 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 977/1958] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:34:44 2022] GET /identity/v3/roles => generated 1420 bytes in 543 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:34:45.300460 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3167258-a08d-45be-9327-c8292ed63953 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:45.302738 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-327f8fc5-eabd-4fd6-a784-577b1b546897 service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:45.303681 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-f3167258-a08d-45be-9327-c8292ed63953 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:45.314243 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 982/1959] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:34:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 141 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:34:45.315237 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-327f8fc5-eabd-4fd6-a784-577b1b546897 service neutron] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:45.317594 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-327f8fc5-eabd-4fd6-a784-577b1b546897 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:45.325561 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3167258-a08d-45be-9327-c8292ed63953 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:45.334263 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76ad1133-a31a-4d79-8e8c-e0af0cc54a57 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:45.335298 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76ad1133-a31a-4d79-8e8c-e0af0cc54a57 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:45.335639 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76ad1133-a31a-4d79-8e8c-e0af0cc54a57 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:45.335926 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76ad1133-a31a-4d79-8e8c-e0af0cc54a57 admin admin] PATH_INFO: `/v3/projects/03d5e71538dc44bcad7a72876877b7b6` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:45.337136 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76ad1133-a31a-4d79-8e8c-e0af0cc54a57 admin admin] RBAC: Authorizing `identity:delete_project(project_id=03d5e71538dc44bcad7a72876877b7b6)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:45.339843 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76ad1133-a31a-4d79-8e8c-e0af0cc54a57 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:45.356415 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-327f8fc5-eabd-4fd6-a784-577b1b546897 service neutron] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:45.364744 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-033621ca-bc41-4df7-8949-79f024ad9685 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:45.366032 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-033621ca-bc41-4df7-8949-79f024ad9685 admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:45.366634 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-033621ca-bc41-4df7-8949-79f024ad9685 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:45.367163 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-033621ca-bc41-4df7-8949-79f024ad9685 admin admin] PATH_INFO: `/v3/projects/b8d69910a0ca48a5b09b918cb326244a/users/0a147ed4877a4e08b9ea10fa4098ed73/roles/a6e562cdc96c4be691d638990f8403fd` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:45.397810 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:DELETE=11,keystone:SELECT=88,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:34:45.415091 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-033621ca-bc41-4df7-8949-79f024ad9685 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b8d69910a0ca48a5b09b918cb326244a, user_id=0a147ed4877a4e08b9ea10fa4098ed73, role_id=a6e562cdc96c4be691d638990f8403fd)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:45.424109 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-033621ca-bc41-4df7-8949-79f024ad9685 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:45.437527 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-76ad1133-a31a-4d79-8e8c-e0af0cc54a57 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '03d5e71538dc44bcad7a72876877b7b6', 'request_id': 'req-76ad1133-a31a-4d79-8e8c-e0af0cc54a57'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:45.454016 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-76ad1133-a31a-4d79-8e8c-e0af0cc54a57 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '03d5e71538dc44bcad7a72876877b7b6', 'request_id': 'req-76ad1133-a31a-4d79-8e8c-e0af0cc54a57'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:45.471163 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-76ad1133-a31a-4d79-8e8c-e0af0cc54a57 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '03d5e71538dc44bcad7a72876877b7b6', 'request_id': 'req-76ad1133-a31a-4d79-8e8c-e0af0cc54a57'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:45.493034 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 983/1960] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:34:45 2022] PUT /identity/v3/projects/b8d69910a0ca48a5b09b918cb326244a/users/0a147ed4877a4e08b9ea10fa4098ed73/roles/a6e562cdc96c4be691d638990f8403fd => generated 0 bytes in 178 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:34:45.503799 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-033621ca-bc41-4df7-8949-79f024ad9685 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:45.510178 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-76ad1133-a31a-4d79-8e8c-e0af0cc54a57 admin admin] The token cache is being invalidate because project 03d5e71538dc44bcad7a72876877b7b6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:34:45.511599 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-76ad1133-a31a-4d79-8e8c-e0af0cc54a57 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:45.512738 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-76ad1133-a31a-4d79-8e8c-e0af0cc54a57 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:45.515733 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 978/1961] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:34:45 2022] DELETE /identity/v3/projects/03d5e71538dc44bcad7a72876877b7b6 => generated 0 bytes in 218 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:34:45.526123 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e9e3f48-1469-40d9-be26-bc51cc9e2fdf None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:45.527467 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e9e3f48-1469-40d9-be26-bc51cc9e2fdf None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:45.529077 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e9e3f48-1469-40d9-be26-bc51cc9e2fdf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:45.574367 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-0e9e3f48-1469-40d9-be26-bc51cc9e2fdf None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:34:45.576312 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-0e9e3f48-1469-40d9-be26-bc51cc9e2fdf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:45.679985 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-033621ca-bc41-4df7-8949-79f024ad9685 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:45.690038 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-337a9209-cee9-4927-a909-690f5bd19dc6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:45.690038 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-337a9209-cee9-4927-a909-690f5bd19dc6 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:45.690038 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-337a9209-cee9-4927-a909-690f5bd19dc6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:45.690038 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-337a9209-cee9-4927-a909-690f5bd19dc6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:45.692784 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-337a9209-cee9-4927-a909-690f5bd19dc6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:45.711432 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 979/1962] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:34:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:34:45.713598 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d95034c-6989-4ad9-bed5-34b477ea7619 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:45.714114 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d95034c-6989-4ad9-bed5-34b477ea7619 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:45.715104 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d95034c-6989-4ad9-bed5-34b477ea7619 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:45.722561 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-337a9209-cee9-4927-a909-690f5bd19dc6 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:45.747894 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-8d95034c-6989-4ad9-bed5-34b477ea7619 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:34:45.750682 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-8d95034c-6989-4ad9-bed5-34b477ea7619 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:45.769851 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 980/1963] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:34:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 54 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 21:34:45.772384 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d95034c-6989-4ad9-bed5-34b477ea7619 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:45.789327 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 984/1964] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:34:45 2022] GET /identity/v3/roles => generated 1420 bytes in 287 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:34:45.790679 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d95034c-6989-4ad9-bed5-34b477ea7619 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:45.792601 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-337a9209-cee9-4927-a909-690f5bd19dc6 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:45.800080 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-337a9209-cee9-4927-a909-690f5bd19dc6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:45.800203 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d4fc784-b24c-48d2-988c-494fbedde20a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:45.800536 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d4fc784-b24c-48d2-988c-494fbedde20a admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:45.801972 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d4fc784-b24c-48d2-988c-494fbedde20a admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:45.802626 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d4fc784-b24c-48d2-988c-494fbedde20a admin admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:45.804887 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d4fc784-b24c-48d2-988c-494fbedde20a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:34:45.805546 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d4fc784-b24c-48d2-988c-494fbedde20a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:45.809178 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d4fc784-b24c-48d2-988c-494fbedde20a admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:45.827863 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-337a9209-cee9-4927-a909-690f5bd19dc6 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:45.844853 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 981/1965] 10.222.0.90 () {58 vars in 1153 bytes} [Mon Aug 29 21:34:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 73 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:34:45.849845 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d4fc784-b24c-48d2-988c-494fbedde20a admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:45.850115 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e2983f5-b68f-46d1-b69a-5387cabd58bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:45.850587 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e2983f5-b68f-46d1-b69a-5387cabd58bc admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:45.850870 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e2983f5-b68f-46d1-b69a-5387cabd58bc admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:45.851135 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e2983f5-b68f-46d1-b69a-5387cabd58bc admin admin] PATH_INFO: `/v3/projects/b8d69910a0ca48a5b09b918cb326244a/users/0a147ed4877a4e08b9ea10fa4098ed73/roles/a6e562cdc96c4be691d638990f8403fd` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:45.863549 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d4fc784-b24c-48d2-988c-494fbedde20a admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:45.864905 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e2983f5-b68f-46d1-b69a-5387cabd58bc admin admin] RBAC: Authorizing `identity:create_grant(project_id=b8d69910a0ca48a5b09b918cb326244a, user_id=0a147ed4877a4e08b9ea10fa4098ed73, role_id=a6e562cdc96c4be691d638990f8403fd)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:45.868042 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e2983f5-b68f-46d1-b69a-5387cabd58bc admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:45.871510 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38190a5f-c4b5-4a66-9a49-eb206884ab9b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:45.872265 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38190a5f-c4b5-4a66-9a49-eb206884ab9b admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:45.872955 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38190a5f-c4b5-4a66-9a49-eb206884ab9b admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:45.873331 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38190a5f-c4b5-4a66-9a49-eb206884ab9b admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:45.874355 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38190a5f-c4b5-4a66-9a49-eb206884ab9b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:45.882152 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38190a5f-c4b5-4a66-9a49-eb206884ab9b admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:45.899907 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 985/1966] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:34:45 2022] PUT /identity/v3/projects/b8d69910a0ca48a5b09b918cb326244a/users/0a147ed4877a4e08b9ea10fa4098ed73/roles/a6e562cdc96c4be691d638990f8403fd => generated 0 bytes in 110 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:34:45.913356 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48549721-665e-4e76-8cfd-e4895a79d136 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:45.914177 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48549721-665e-4e76-8cfd-e4895a79d136 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:45.914561 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48549721-665e-4e76-8cfd-e4895a79d136 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:45.958741 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 982/1967] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:34:45 2022] POST /identity/v3/projects => generated 381 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 21:34:45.960380 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38190a5f-c4b5-4a66-9a49-eb206884ab9b admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:45.964994 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-38190a5f-c4b5-4a66-9a49-eb206884ab9b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:45.983463 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-48549721-665e-4e76-8cfd-e4895a79d136 None None] MFA Rules not processed for user `0a147ed4877a4e08b9ea10fa4098ed73`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:34:45.985113 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-48549721-665e-4e76-8cfd-e4895a79d136 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:46.058122 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:34:46.100527 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38190a5f-c4b5-4a66-9a49-eb206884ab9b admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:46.106574 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25c37f24-7095-45d9-8dea-036824ea0898 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:46.107524 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25c37f24-7095-45d9-8dea-036824ea0898 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:46.108070 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25c37f24-7095-45d9-8dea-036824ea0898 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:46.108434 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25c37f24-7095-45d9-8dea-036824ea0898 admin admin] PATH_INFO: `/v3/users/858dff9823754bae941f3cc2e957052c` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:46.109919 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25c37f24-7095-45d9-8dea-036824ea0898 admin admin] RBAC: Authorizing `identity:delete_user(user_id=858dff9823754bae941f3cc2e957052c)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:46.112182 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25c37f24-7095-45d9-8dea-036824ea0898 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:46.114336 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 986/1968] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:34:45 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 205 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:34:46.116320 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48549721-665e-4e76-8cfd-e4895a79d136 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:46.136070 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48549721-665e-4e76-8cfd-e4895a79d136 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:46.141884 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8627867-73a8-41cd-8070-c7fa7f8fe057 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:46.142770 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:DELETE=7,keystone:SELECT=91,keystone:INSERT=2 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:34:46.142967 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8627867-73a8-41cd-8070-c7fa7f8fe057 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:46.143360 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8627867-73a8-41cd-8070-c7fa7f8fe057 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:46.143797 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8627867-73a8-41cd-8070-c7fa7f8fe057 admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:46.144860 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8627867-73a8-41cd-8070-c7fa7f8fe057 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:46.148827 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8627867-73a8-41cd-8070-c7fa7f8fe057 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:46.187591 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 987/1969] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:34:46 2022] POST /identity/v3/users => generated 291 bytes in 73 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:34:46.188677 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8627867-73a8-41cd-8070-c7fa7f8fe057 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:46.191201 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-f8627867-73a8-41cd-8070-c7fa7f8fe057 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:46.210491 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-25c37f24-7095-45d9-8dea-036824ea0898 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '858dff9823754bae941f3cc2e957052c'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:46.212181 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8627867-73a8-41cd-8070-c7fa7f8fe057 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:46.220657 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4599638c-eec2-4ed5-a39f-9eefe7617baa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:46.221551 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4599638c-eec2-4ed5-a39f-9eefe7617baa admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:46.222095 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4599638c-eec2-4ed5-a39f-9eefe7617baa admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:46.222554 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4599638c-eec2-4ed5-a39f-9eefe7617baa admin admin] PATH_INFO: `/v3/projects/5b020c39ae3f4d1fba39663bc5d05978` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:46.224718 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4599638c-eec2-4ed5-a39f-9eefe7617baa admin admin] RBAC: Authorizing `identity:delete_project(project_id=5b020c39ae3f4d1fba39663bc5d05978)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:46.228460 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4599638c-eec2-4ed5-a39f-9eefe7617baa admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:46.245185 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-25c37f24-7095-45d9-8dea-036824ea0898 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '858dff9823754bae941f3cc2e957052c'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:46.246360 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-25c37f24-7095-45d9-8dea-036824ea0898 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '858dff9823754bae941f3cc2e957052c'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:46.251969 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-25c37f24-7095-45d9-8dea-036824ea0898 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '858dff9823754bae941f3cc2e957052c'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:46.269162 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-25c37f24-7095-45d9-8dea-036824ea0898 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '858dff9823754bae941f3cc2e957052c'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:46.296760 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 983/1970] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:34:45 2022] DELETE /identity/v3/users/858dff9823754bae941f3cc2e957052c => generated 0 bytes in 337 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:34:46.298376 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25c37f24-7095-45d9-8dea-036824ea0898 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:46.301056 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-25c37f24-7095-45d9-8dea-036824ea0898 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:46.325558 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-4599638c-eec2-4ed5-a39f-9eefe7617baa admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5b020c39ae3f4d1fba39663bc5d05978', 'request_id': 'req-4599638c-eec2-4ed5-a39f-9eefe7617baa'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:46.336948 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-4599638c-eec2-4ed5-a39f-9eefe7617baa admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5b020c39ae3f4d1fba39663bc5d05978', 'request_id': 'req-4599638c-eec2-4ed5-a39f-9eefe7617baa'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:46.353852 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-4599638c-eec2-4ed5-a39f-9eefe7617baa admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5b020c39ae3f4d1fba39663bc5d05978', 'request_id': 'req-4599638c-eec2-4ed5-a39f-9eefe7617baa'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:46.379279 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-4599638c-eec2-4ed5-a39f-9eefe7617baa admin admin] The token cache is being invalidate because project 5b020c39ae3f4d1fba39663bc5d05978 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81434) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:34:46.380128 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-4599638c-eec2-4ed5-a39f-9eefe7617baa admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:46.380997 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-4599638c-eec2-4ed5-a39f-9eefe7617baa admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:46.383788 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 988/1971] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:34:46 2022] DELETE /identity/v3/projects/5b020c39ae3f4d1fba39663bc5d05978 => generated 0 bytes in 195 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:34:46.384135 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4599638c-eec2-4ed5-a39f-9eefe7617baa admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:46.385617 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-4599638c-eec2-4ed5-a39f-9eefe7617baa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:46.420531 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25c37f24-7095-45d9-8dea-036824ea0898 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:46.424609 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dd27733-f9e2-4624-b4ec-6401f2fc7fca service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dc226f7ddcb49ecacb51ce57fdfc11b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:46.425421 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dd27733-f9e2-4624-b4ec-6401f2fc7fca service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:46.425777 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dd27733-f9e2-4624-b4ec-6401f2fc7fca service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:46.426158 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dd27733-f9e2-4624-b4ec-6401f2fc7fca service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:46.429171 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-0dd27733-f9e2-4624-b4ec-6401f2fc7fca service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:46.515122 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4599638c-eec2-4ed5-a39f-9eefe7617baa admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:46.536287 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dd27733-f9e2-4624-b4ec-6401f2fc7fca service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:46.537865 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dd27733-f9e2-4624-b4ec-6401f2fc7fca service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:46.548750 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 984/1972] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:34:46 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 251 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:34:46.550129 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3272ab45-a479-42f7-a747-04e262c8f7fc None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:46.550443 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3272ab45-a479-42f7-a747-04e262c8f7fc None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:46.550737 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3272ab45-a479-42f7-a747-04e262c8f7fc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:46.573458 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-3272ab45-a479-42f7-a747-04e262c8f7fc None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:34:46.575621 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-3272ab45-a479-42f7-a747-04e262c8f7fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:46.593680 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=89,keystone:INSERT=4,keystone:DELETE=7 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:34:46.629967 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd8d29f0-9607-4dca-8128-b09eabb993c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:46.631659 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd8d29f0-9607-4dca-8128-b09eabb993c6 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:46.632330 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd8d29f0-9607-4dca-8128-b09eabb993c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:46.632879 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd8d29f0-9607-4dca-8128-b09eabb993c6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:46.634153 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd8d29f0-9607-4dca-8128-b09eabb993c6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:46.637149 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd8d29f0-9607-4dca-8128-b09eabb993c6 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:46.648792 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:34:46.669870 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 989/1973] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:34:46 2022] GET /identity/v3/roles => generated 1420 bytes in 284 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:34:46.671774 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd8d29f0-9607-4dca-8128-b09eabb993c6 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:46.678266 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-fd8d29f0-9607-4dca-8128-b09eabb993c6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:46.687185 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 985/1974] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:34:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:34:46.687185 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3272ab45-a479-42f7-a747-04e262c8f7fc None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:46.705233 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd8d29f0-9607-4dca-8128-b09eabb993c6 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:46.708562 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3272ab45-a479-42f7-a747-04e262c8f7fc None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:46.712844 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a919af3e-d865-40f0-9c0d-3ed5705ec83a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:46.713895 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a919af3e-d865-40f0-9c0d-3ed5705ec83a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:46.714459 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a919af3e-d865-40f0-9c0d-3ed5705ec83a admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:46.714967 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a919af3e-d865-40f0-9c0d-3ed5705ec83a admin admin] PATH_INFO: `/v3/users/d115ddd16b73493691b2a697bd4fac84` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:46.716662 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a919af3e-d865-40f0-9c0d-3ed5705ec83a admin admin] RBAC: Authorizing `identity:delete_user(user_id=d115ddd16b73493691b2a697bd4fac84)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:46.717332 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa1097df-3a45-4234-b979-f84b852b4026 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:46.718414 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa1097df-3a45-4234-b979-f84b852b4026 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:46.718745 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa1097df-3a45-4234-b979-f84b852b4026 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:46.719103 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa1097df-3a45-4234-b979-f84b852b4026 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:46.719838 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a919af3e-d865-40f0-9c0d-3ed5705ec83a admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:46.720011 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa1097df-3a45-4234-b979-f84b852b4026 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:46.721757 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa1097df-3a45-4234-b979-f84b852b4026 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:46.745052 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 986/1975] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:34:46 2022] GET /identity/v3/roles => generated 1420 bytes in 63 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:34:46.746877 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa1097df-3a45-4234-b979-f84b852b4026 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:46.751059 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-fa1097df-3a45-4234-b979-f84b852b4026 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:46.807047 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-a919af3e-d865-40f0-9c0d-3ed5705ec83a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd115ddd16b73493691b2a697bd4fac84'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:46.816358 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-a919af3e-d865-40f0-9c0d-3ed5705ec83a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd115ddd16b73493691b2a697bd4fac84'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:46.835662 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-a919af3e-d865-40f0-9c0d-3ed5705ec83a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd115ddd16b73493691b2a697bd4fac84'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:46.836960 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-a919af3e-d865-40f0-9c0d-3ed5705ec83a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd115ddd16b73493691b2a697bd4fac84'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:46.839073 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-a919af3e-d865-40f0-9c0d-3ed5705ec83a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd115ddd16b73493691b2a697bd4fac84'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:46.874432 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 990/1976] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:34:46 2022] DELETE /identity/v3/users/d115ddd16b73493691b2a697bd4fac84 => generated 0 bytes in 199 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:34:46.874432 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a919af3e-d865-40f0-9c0d-3ed5705ec83a admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:46.874432 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-a919af3e-d865-40f0-9c0d-3ed5705ec83a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:46.881763 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa1097df-3a45-4234-b979-f84b852b4026 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:47.053133 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be72fb63-2d3d-4974-9d6d-0467cb1fa38b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:47.054384 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be72fb63-2d3d-4974-9d6d-0467cb1fa38b service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:47.054897 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be72fb63-2d3d-4974-9d6d-0467cb1fa38b service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:47.055462 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be72fb63-2d3d-4974-9d6d-0467cb1fa38b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:47.059163 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-be72fb63-2d3d-4974-9d6d-0467cb1fa38b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:47.073992 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a919af3e-d865-40f0-9c0d-3ed5705ec83a admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:47.086159 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39021ee7-45f5-40a8-8625-3e38369174fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:47.086159 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39021ee7-45f5-40a8-8625-3e38369174fd admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:47.086159 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39021ee7-45f5-40a8-8625-3e38369174fd admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:47.086159 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39021ee7-45f5-40a8-8625-3e38369174fd admin admin] PATH_INFO: `/v3/projects/b0e9b513af904caf84105db15cd77dda/users/878148ec3d3a4ecba9f7dfa55e1bc27e/roles/02e242d5520740d4b77a5f4008c7eb57` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:47.086810 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be72fb63-2d3d-4974-9d6d-0467cb1fa38b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:47.088054 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be72fb63-2d3d-4974-9d6d-0467cb1fa38b service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:47.111789 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39021ee7-45f5-40a8-8625-3e38369174fd admin admin] RBAC: Authorizing `identity:create_grant(project_id=b0e9b513af904caf84105db15cd77dda, user_id=878148ec3d3a4ecba9f7dfa55e1bc27e, role_id=02e242d5520740d4b77a5f4008c7eb57)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:47.115027 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 987/1977] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:34:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 369 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:34:47.123811 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39021ee7-45f5-40a8-8625-3e38369174fd admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:47.167023 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 991/1978] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:34:46 2022] PUT /identity/v3/projects/b0e9b513af904caf84105db15cd77dda/users/878148ec3d3a4ecba9f7dfa55e1bc27e/roles/02e242d5520740d4b77a5f4008c7eb57 => generated 0 bytes in 289 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:34:47.173758 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f35bf89-4a45-4dd5-a8ef-d887b532d76a None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:47.174054 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f35bf89-4a45-4dd5-a8ef-d887b532d76a None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:47.174330 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f35bf89-4a45-4dd5-a8ef-d887b532d76a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:47.231055 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-0f35bf89-4a45-4dd5-a8ef-d887b532d76a None None] MFA Rules not processed for user `878148ec3d3a4ecba9f7dfa55e1bc27e`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:34:47.234849 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-0f35bf89-4a45-4dd5-a8ef-d887b532d76a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:47.391064 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 988/1979] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:34:47 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 219 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:34:47.398467 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39021ee7-45f5-40a8-8625-3e38369174fd admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:47.447720 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f35bf89-4a45-4dd5-a8ef-d887b532d76a None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:47.451615 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-0f35bf89-4a45-4dd5-a8ef-d887b532d76a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:47.493117 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:34:47.494577 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:34:47.555462 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39021ee7-45f5-40a8-8625-3e38369174fd admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:47.565026 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da97092d-aeec-4f7c-b081-f5d2d11a3e3e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:47.565026 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da97092d-aeec-4f7c-b081-f5d2d11a3e3e admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:47.565506 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da97092d-aeec-4f7c-b081-f5d2d11a3e3e admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:47.566032 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da97092d-aeec-4f7c-b081-f5d2d11a3e3e admin admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:47.567801 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da97092d-aeec-4f7c-b081-f5d2d11a3e3e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:47.571642 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da97092d-aeec-4f7c-b081-f5d2d11a3e3e admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:47.599497 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f35bf89-4a45-4dd5-a8ef-d887b532d76a None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:47.606968 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfc11320-b847-4104-ad33-575bd6d29f96 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:47.607921 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfc11320-b847-4104-ad33-575bd6d29f96 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:47.610418 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfc11320-b847-4104-ad33-575bd6d29f96 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:47.611107 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfc11320-b847-4104-ad33-575bd6d29f96 admin admin] PATH_INFO: `/v3/projects/a485a9d3f22046acb8cf932f589de51d` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:47.613163 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfc11320-b847-4104-ad33-575bd6d29f96 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a485a9d3f22046acb8cf932f589de51d)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:47.616776 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfc11320-b847-4104-ad33-575bd6d29f96 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:47.650355 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 992/1980] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:34:47 2022] POST /identity/v3/projects => generated 381 bytes in 253 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:34:47.651986 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da97092d-aeec-4f7c-b081-f5d2d11a3e3e admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:47.655280 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-da97092d-aeec-4f7c-b081-f5d2d11a3e3e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:47.710400 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-cfc11320-b847-4104-ad33-575bd6d29f96 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a485a9d3f22046acb8cf932f589de51d', 'request_id': 'req-cfc11320-b847-4104-ad33-575bd6d29f96'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:47.720004 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-cfc11320-b847-4104-ad33-575bd6d29f96 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a485a9d3f22046acb8cf932f589de51d', 'request_id': 'req-cfc11320-b847-4104-ad33-575bd6d29f96'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:47.729256 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-cfc11320-b847-4104-ad33-575bd6d29f96 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a485a9d3f22046acb8cf932f589de51d', 'request_id': 'req-cfc11320-b847-4104-ad33-575bd6d29f96'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:47.765131 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-cfc11320-b847-4104-ad33-575bd6d29f96 admin admin] The token cache is being invalidate because project a485a9d3f22046acb8cf932f589de51d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:34:47.765986 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-cfc11320-b847-4104-ad33-575bd6d29f96 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:47.766911 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-cfc11320-b847-4104-ad33-575bd6d29f96 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:47.769246 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 989/1981] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:34:47 2022] DELETE /identity/v3/projects/a485a9d3f22046acb8cf932f589de51d => generated 0 bytes in 322 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 21:34:47.771177 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfc11320-b847-4104-ad33-575bd6d29f96 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:47.773201 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-cfc11320-b847-4104-ad33-575bd6d29f96 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:47.840831 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da97092d-aeec-4f7c-b081-f5d2d11a3e3e admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:47.891275 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfc11320-b847-4104-ad33-575bd6d29f96 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:47.896945 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b57a463-4e69-4e25-b7cf-ebbebeb908bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:47.898001 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b57a463-4e69-4e25-b7cf-ebbebeb908bc admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:47.898509 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b57a463-4e69-4e25-b7cf-ebbebeb908bc admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:47.899021 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b57a463-4e69-4e25-b7cf-ebbebeb908bc admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:47.900385 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b57a463-4e69-4e25-b7cf-ebbebeb908bc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:47.903937 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b57a463-4e69-4e25-b7cf-ebbebeb908bc admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:47.972585 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 990/1982] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:34:47 2022] POST /identity/v3/users => generated 297 bytes in 200 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:34:47.972585 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b57a463-4e69-4e25-b7cf-ebbebeb908bc admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:47.972866 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-5b57a463-4e69-4e25-b7cf-ebbebeb908bc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:47.987238 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b57a463-4e69-4e25-b7cf-ebbebeb908bc admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:47.991945 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-997b39cb-0041-4b5d-86d6-ca22c6d6ef83 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:47.992641 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-997b39cb-0041-4b5d-86d6-ca22c6d6ef83 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:47.992824 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8990269-8ee4-4133-99c4-76af61b41023 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:47.993196 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-997b39cb-0041-4b5d-86d6-ca22c6d6ef83 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:47.993431 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-997b39cb-0041-4b5d-86d6-ca22c6d6ef83 admin admin] PATH_INFO: `/v3/users/ba6cbf68ca7245cdaeb1cc6975b23d0d` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:47.993909 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8990269-8ee4-4133-99c4-76af61b41023 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:47.994386 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8990269-8ee4-4133-99c4-76af61b41023 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:47.994833 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8990269-8ee4-4133-99c4-76af61b41023 admin admin] PATH_INFO: `/v3/projects/c04c6210dab04f3d9a6cf79517956a25` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:47.994994 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-997b39cb-0041-4b5d-86d6-ca22c6d6ef83 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ba6cbf68ca7245cdaeb1cc6975b23d0d)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:47.996755 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8990269-8ee4-4133-99c4-76af61b41023 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c04c6210dab04f3d9a6cf79517956a25)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:47.996921 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-997b39cb-0041-4b5d-86d6-ca22c6d6ef83 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:47.999819 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8990269-8ee4-4133-99c4-76af61b41023 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:48.063929 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-997b39cb-0041-4b5d-86d6-ca22c6d6ef83 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ba6cbf68ca7245cdaeb1cc6975b23d0d'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:48.070091 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=87,keystone:DELETE=9,keystone:INSERT=4 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:34:48.086866 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-997b39cb-0041-4b5d-86d6-ca22c6d6ef83 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ba6cbf68ca7245cdaeb1cc6975b23d0d'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:48.092003 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-c8990269-8ee4-4133-99c4-76af61b41023 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c04c6210dab04f3d9a6cf79517956a25', 'request_id': 'req-c8990269-8ee4-4133-99c4-76af61b41023'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:48.100816 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-997b39cb-0041-4b5d-86d6-ca22c6d6ef83 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ba6cbf68ca7245cdaeb1cc6975b23d0d'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:48.101858 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-997b39cb-0041-4b5d-86d6-ca22c6d6ef83 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ba6cbf68ca7245cdaeb1cc6975b23d0d'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:48.103021 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-997b39cb-0041-4b5d-86d6-ca22c6d6ef83 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ba6cbf68ca7245cdaeb1cc6975b23d0d'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:48.104222 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-c8990269-8ee4-4133-99c4-76af61b41023 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c04c6210dab04f3d9a6cf79517956a25', 'request_id': 'req-c8990269-8ee4-4133-99c4-76af61b41023'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:48.118230 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=89,keystone:INSERT=2,keystone:DELETE=9 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:34:48.123296 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-c8990269-8ee4-4133-99c4-76af61b41023 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c04c6210dab04f3d9a6cf79517956a25', 'request_id': 'req-c8990269-8ee4-4133-99c4-76af61b41023'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:48.139148 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 993/1983] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:34:47 2022] DELETE /identity/v3/users/ba6cbf68ca7245cdaeb1cc6975b23d0d => generated 0 bytes in 488 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:34:48.141071 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b944805-c23e-421e-b1af-e4579c9be015 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:48.141579 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b944805-c23e-421e-b1af-e4579c9be015 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:48.142119 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b944805-c23e-421e-b1af-e4579c9be015 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:48.152800 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-c8990269-8ee4-4133-99c4-76af61b41023 admin admin] The token cache is being invalidate because project c04c6210dab04f3d9a6cf79517956a25 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:34:48.153181 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-c8990269-8ee4-4133-99c4-76af61b41023 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:48.154106 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-c8990269-8ee4-4133-99c4-76af61b41023 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:48.155815 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 991/1984] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:34:47 2022] DELETE /identity/v3/projects/c04c6210dab04f3d9a6cf79517956a25 => generated 0 bytes in 186 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:34:48.156578 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8990269-8ee4-4133-99c4-76af61b41023 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:48.158089 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-c8990269-8ee4-4133-99c4-76af61b41023 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:48.174733 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-8b944805-c23e-421e-b1af-e4579c9be015 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:34:48.176661 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-8b944805-c23e-421e-b1af-e4579c9be015 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:48.257318 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 994/1985] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:34:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:34:48.259503 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeabe636-016d-41eb-addd-776676aac35e None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:48.259986 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeabe636-016d-41eb-addd-776676aac35e None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:48.260453 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeabe636-016d-41eb-addd-776676aac35e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:48.273478 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8990269-8ee4-4133-99c4-76af61b41023 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:48.282852 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a9077fe-4e99-4daf-9a56-f299d3ff4670 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:48.283720 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a9077fe-4e99-4daf-9a56-f299d3ff4670 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:48.284094 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a9077fe-4e99-4daf-9a56-f299d3ff4670 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:48.284713 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a9077fe-4e99-4daf-9a56-f299d3ff4670 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:48.285640 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a9077fe-4e99-4daf-9a56-f299d3ff4670 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:48.288550 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a9077fe-4e99-4daf-9a56-f299d3ff4670 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:48.293667 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-aeabe636-016d-41eb-addd-776676aac35e None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:34:48.295925 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-aeabe636-016d-41eb-addd-776676aac35e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:48.314136 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 992/1986] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:34:48 2022] GET /identity/v3/roles => generated 1420 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:34:48.314386 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 995/1987] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:34:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 53 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:34:48.314539 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a9077fe-4e99-4daf-9a56-f299d3ff4670 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:48.320461 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19a2aa7a-e7ba-4f34-b9b5-5958daede515 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:48.320461 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19a2aa7a-e7ba-4f34-b9b5-5958daede515 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:48.320461 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19a2aa7a-e7ba-4f34-b9b5-5958daede515 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:48.321331 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-8a9077fe-4e99-4daf-9a56-f299d3ff4670 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:48.346477 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a9077fe-4e99-4daf-9a56-f299d3ff4670 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:48.355510 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-19a2aa7a-e7ba-4f34-b9b5-5958daede515 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:34:48.356797 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-19a2aa7a-e7ba-4f34-b9b5-5958daede515 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:48.357386 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0be5cdac-55a9-4150-befb-8aae4334f04a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:48.358292 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0be5cdac-55a9-4150-befb-8aae4334f04a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:48.358802 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0be5cdac-55a9-4150-befb-8aae4334f04a admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:48.359308 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0be5cdac-55a9-4150-befb-8aae4334f04a admin admin] PATH_INFO: `/v3/users/f91680f344214808954493dc7e276a5f` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:48.361269 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0be5cdac-55a9-4150-befb-8aae4334f04a admin admin] RBAC: Authorizing `identity:delete_user(user_id=f91680f344214808954493dc7e276a5f)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:48.365172 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0be5cdac-55a9-4150-befb-8aae4334f04a admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:48.372147 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 996/1988] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:34:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 60 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:34:48.372944 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19a2aa7a-e7ba-4f34-b9b5-5958daede515 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:48.400647 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19a2aa7a-e7ba-4f34-b9b5-5958daede515 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:48.407306 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99d01eb0-07b8-4e8f-bc27-781c6ea4aa73 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:48.408179 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99d01eb0-07b8-4e8f-bc27-781c6ea4aa73 admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:48.408559 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99d01eb0-07b8-4e8f-bc27-781c6ea4aa73 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:48.408872 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99d01eb0-07b8-4e8f-bc27-781c6ea4aa73 admin admin] PATH_INFO: `/v3/projects/cb3bc5833ea94b3bacca8826b813b662/users/572ed224ebbc4e558a4c7c2ce8201bd5/roles/a6e562cdc96c4be691d638990f8403fd` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:48.424879 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99d01eb0-07b8-4e8f-bc27-781c6ea4aa73 admin admin] RBAC: Authorizing `identity:create_grant(project_id=cb3bc5833ea94b3bacca8826b813b662, user_id=572ed224ebbc4e558a4c7c2ce8201bd5, role_id=a6e562cdc96c4be691d638990f8403fd)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:48.427851 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99d01eb0-07b8-4e8f-bc27-781c6ea4aa73 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:48.432029 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-0be5cdac-55a9-4150-befb-8aae4334f04a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f91680f344214808954493dc7e276a5f'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:48.442500 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 997/1989] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:34:48 2022] PUT /identity/v3/projects/cb3bc5833ea94b3bacca8826b813b662/users/572ed224ebbc4e558a4c7c2ce8201bd5/roles/a6e562cdc96c4be691d638990f8403fd => generated 0 bytes in 70 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:34:48.443620 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99d01eb0-07b8-4e8f-bc27-781c6ea4aa73 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:48.445270 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-99d01eb0-07b8-4e8f-bc27-781c6ea4aa73 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:48.452831 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-0be5cdac-55a9-4150-befb-8aae4334f04a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f91680f344214808954493dc7e276a5f'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:48.454006 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-0be5cdac-55a9-4150-befb-8aae4334f04a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f91680f344214808954493dc7e276a5f'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:48.455106 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-0be5cdac-55a9-4150-befb-8aae4334f04a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f91680f344214808954493dc7e276a5f'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:48.469185 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-0be5cdac-55a9-4150-befb-8aae4334f04a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f91680f344214808954493dc7e276a5f'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:48.487111 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 993/1990] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:34:48 2022] DELETE /identity/v3/users/f91680f344214808954493dc7e276a5f => generated 0 bytes in 177 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:34:48.487996 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0be5cdac-55a9-4150-befb-8aae4334f04a admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:48.490803 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-0be5cdac-55a9-4150-befb-8aae4334f04a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:48.525508 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99d01eb0-07b8-4e8f-bc27-781c6ea4aa73 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:48.544477 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:DELETE=15,keystone:SELECT=83,keystone:INSERT=2 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:34:48.600758 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0be5cdac-55a9-4150-befb-8aae4334f04a admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:48.611434 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7027fa7-0739-4e61-b17d-5b7bc09a0b0a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:48.612791 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7027fa7-0739-4e61-b17d-5b7bc09a0b0a admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:48.613361 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7027fa7-0739-4e61-b17d-5b7bc09a0b0a admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:48.613697 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5918d107-2557-4792-a51b-02f580a05db1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:48.614553 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7027fa7-0739-4e61-b17d-5b7bc09a0b0a admin admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:48.615414 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5918d107-2557-4792-a51b-02f580a05db1 service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:48.615854 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5918d107-2557-4792-a51b-02f580a05db1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:48.616266 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5918d107-2557-4792-a51b-02f580a05db1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:48.616826 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7027fa7-0739-4e61-b17d-5b7bc09a0b0a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:34:48.617373 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7027fa7-0739-4e61-b17d-5b7bc09a0b0a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:48.618748 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-5918d107-2557-4792-a51b-02f580a05db1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:48.621910 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7027fa7-0739-4e61-b17d-5b7bc09a0b0a admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:48.635618 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5918d107-2557-4792-a51b-02f580a05db1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:48.640006 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5918d107-2557-4792-a51b-02f580a05db1 service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:48.646364 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 994/1991] 10.222.0.90 () {58 vars in 1153 bytes} [Mon Aug 29 21:34:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:34:48.646364 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7027fa7-0739-4e61-b17d-5b7bc09a0b0a admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:48.652023 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-c7027fa7-0739-4e61-b17d-5b7bc09a0b0a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:48.660529 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 998/1992] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:34:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 217 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:34:48.663440 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5918d107-2557-4792-a51b-02f580a05db1 service neutron] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:48.672953 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7027fa7-0739-4e61-b17d-5b7bc09a0b0a admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:48.679615 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00f76045-9b4e-48a4-9002-15352fcac7b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:48.680382 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00f76045-9b4e-48a4-9002-15352fcac7b2 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:48.680736 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00f76045-9b4e-48a4-9002-15352fcac7b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:48.681089 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00f76045-9b4e-48a4-9002-15352fcac7b2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:48.681837 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00f76045-9b4e-48a4-9002-15352fcac7b2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:48.683560 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5918d107-2557-4792-a51b-02f580a05db1 service neutron] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:48.683890 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00f76045-9b4e-48a4-9002-15352fcac7b2 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:48.694645 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-587b65b0-6e07-49f9-8fa6-2af21d0e07dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:48.695606 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-587b65b0-6e07-49f9-8fa6-2af21d0e07dd admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:48.696118 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-587b65b0-6e07-49f9-8fa6-2af21d0e07dd admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:48.696633 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-587b65b0-6e07-49f9-8fa6-2af21d0e07dd admin admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:48.697673 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-587b65b0-6e07-49f9-8fa6-2af21d0e07dd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:48.700748 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-587b65b0-6e07-49f9-8fa6-2af21d0e07dd admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:48.710218 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 995/1993] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:34:48 2022] GET /identity/v3/roles => generated 1420 bytes in 67 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:34:48.715367 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00f76045-9b4e-48a4-9002-15352fcac7b2 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:48.719189 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=98,keystone:DELETE=1,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:34:48.734216 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00f76045-9b4e-48a4-9002-15352fcac7b2 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:48.744972 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-494bd095-616f-47f9-b7aa-530d4ded2dbe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:48.746634 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-494bd095-616f-47f9-b7aa-530d4ded2dbe admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:48.750114 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-494bd095-616f-47f9-b7aa-530d4ded2dbe admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:48.750943 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-494bd095-616f-47f9-b7aa-530d4ded2dbe admin admin] PATH_INFO: `/v3/projects/cb3bc5833ea94b3bacca8826b813b662/users/572ed224ebbc4e558a4c7c2ce8201bd5/roles/a6e562cdc96c4be691d638990f8403fd` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:48.758440 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-494bd095-616f-47f9-b7aa-530d4ded2dbe admin admin] RBAC: Authorizing `identity:create_grant(project_id=cb3bc5833ea94b3bacca8826b813b662, user_id=572ed224ebbc4e558a4c7c2ce8201bd5, role_id=a6e562cdc96c4be691d638990f8403fd)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:48.772881 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-494bd095-616f-47f9-b7aa-530d4ded2dbe admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:48.774478 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 999/1994] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:34:48 2022] POST /identity/v3/projects => generated 367 bytes in 113 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:34:48.777049 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-587b65b0-6e07-49f9-8fa6-2af21d0e07dd admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:48.803228 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 996/1995] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:34:48 2022] PUT /identity/v3/projects/cb3bc5833ea94b3bacca8826b813b662/users/572ed224ebbc4e558a4c7c2ce8201bd5/roles/a6e562cdc96c4be691d638990f8403fd => generated 0 bytes in 80 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:34:48.808861 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-942e413b-1f0f-4a9d-b9a5-f6af14b6b8be None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:48.809364 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-942e413b-1f0f-4a9d-b9a5-f6af14b6b8be None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:48.814240 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-942e413b-1f0f-4a9d-b9a5-f6af14b6b8be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:48.888853 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-942e413b-1f0f-4a9d-b9a5-f6af14b6b8be None None] MFA Rules not processed for user `572ed224ebbc4e558a4c7c2ce8201bd5`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:34:48.890975 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-942e413b-1f0f-4a9d-b9a5-f6af14b6b8be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:48.978056 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-587b65b0-6e07-49f9-8fa6-2af21d0e07dd admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:49.096587 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 997/1996] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:34:48 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 289 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 21:34:49.097444 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-942e413b-1f0f-4a9d-b9a5-f6af14b6b8be None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:49.099744 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-942e413b-1f0f-4a9d-b9a5-f6af14b6b8be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:49.212844 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42a7ec82-2743-4f56-ade4-cac47cbdd1d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:49.213454 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42a7ec82-2743-4f56-ade4-cac47cbdd1d2 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:49.213766 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42a7ec82-2743-4f56-ade4-cac47cbdd1d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:49.214103 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42a7ec82-2743-4f56-ade4-cac47cbdd1d2 admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:49.215364 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42a7ec82-2743-4f56-ade4-cac47cbdd1d2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:49.217316 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42a7ec82-2743-4f56-ade4-cac47cbdd1d2 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:49.248377 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1000/1997] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:34:48 2022] POST /identity/v3/users => generated 284 bytes in 473 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:34:49.249539 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42a7ec82-2743-4f56-ade4-cac47cbdd1d2 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:49.251827 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-42a7ec82-2743-4f56-ade4-cac47cbdd1d2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:49.284253 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-942e413b-1f0f-4a9d-b9a5-f6af14b6b8be None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:49.296362 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-686197ec-49fe-421a-8e95-6c68ee25f5fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:49.297662 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-686197ec-49fe-421a-8e95-6c68ee25f5fa admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:49.298133 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-686197ec-49fe-421a-8e95-6c68ee25f5fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:49.298577 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-686197ec-49fe-421a-8e95-6c68ee25f5fa admin admin] PATH_INFO: `/v3/projects/3fa41cc81c4147c287a44d7cefa6e77b` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:49.299969 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-686197ec-49fe-421a-8e95-6c68ee25f5fa admin admin] RBAC: Authorizing `identity:delete_project(project_id=3fa41cc81c4147c287a44d7cefa6e77b)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:49.302943 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-686197ec-49fe-421a-8e95-6c68ee25f5fa admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:49.439074 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=97,keystone:INSERT=1,keystone:DELETE=2 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:34:49.477584 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42a7ec82-2743-4f56-ade4-cac47cbdd1d2 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:49.488169 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-686197ec-49fe-421a-8e95-6c68ee25f5fa admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3fa41cc81c4147c287a44d7cefa6e77b', 'request_id': 'req-686197ec-49fe-421a-8e95-6c68ee25f5fa'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:49.501047 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-686197ec-49fe-421a-8e95-6c68ee25f5fa admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3fa41cc81c4147c287a44d7cefa6e77b', 'request_id': 'req-686197ec-49fe-421a-8e95-6c68ee25f5fa'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:49.513558 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-686197ec-49fe-421a-8e95-6c68ee25f5fa admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3fa41cc81c4147c287a44d7cefa6e77b', 'request_id': 'req-686197ec-49fe-421a-8e95-6c68ee25f5fa'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:49.542572 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-686197ec-49fe-421a-8e95-6c68ee25f5fa admin admin] The token cache is being invalidate because project 3fa41cc81c4147c287a44d7cefa6e77b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:34:49.543091 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-686197ec-49fe-421a-8e95-6c68ee25f5fa admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:49.544152 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-686197ec-49fe-421a-8e95-6c68ee25f5fa admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:49.547365 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 998/1998] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:34:49 2022] DELETE /identity/v3/projects/3fa41cc81c4147c287a44d7cefa6e77b => generated 0 bytes in 450 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:34:49.548712 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-686197ec-49fe-421a-8e95-6c68ee25f5fa admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:49.553334 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-686197ec-49fe-421a-8e95-6c68ee25f5fa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:49.565497 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:34:49.577861 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f703ae0c-a621-464f-a20f-78ad12864daf service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dc226f7ddcb49ecacb51ce57fdfc11b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:49.578948 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f703ae0c-a621-464f-a20f-78ad12864daf service cinder] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:49.579506 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f703ae0c-a621-464f-a20f-78ad12864daf service cinder] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:49.580009 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f703ae0c-a621-464f-a20f-78ad12864daf service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:49.583272 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-f703ae0c-a621-464f-a20f-78ad12864daf service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:49.663065 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f703ae0c-a621-464f-a20f-78ad12864daf service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:49.664403 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f703ae0c-a621-464f-a20f-78ad12864daf service cinder] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:49.677004 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1001/1999] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:34:49 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 428 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:34:49.677551 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-686197ec-49fe-421a-8e95-6c68ee25f5fa admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:49.678974 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f703ae0c-a621-464f-a20f-78ad12864daf service cinder] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:49.680702 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-f703ae0c-a621-464f-a20f-78ad12864daf service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:49.686754 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88928226-0a48-4066-b7c5-a30bf32d925e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:49.687869 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88928226-0a48-4066-b7c5-a30bf32d925e admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:49.688530 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88928226-0a48-4066-b7c5-a30bf32d925e admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:49.689005 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88928226-0a48-4066-b7c5-a30bf32d925e admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:49.690181 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88928226-0a48-4066-b7c5-a30bf32d925e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:49.693798 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88928226-0a48-4066-b7c5-a30bf32d925e admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:49.699179 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f703ae0c-a621-464f-a20f-78ad12864daf service cinder] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:49.709267 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6f3e20d-f561-495f-abc1-b4d54d52a0dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:49.709728 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6f3e20d-f561-495f-abc1-b4d54d52a0dc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:49.710137 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6f3e20d-f561-495f-abc1-b4d54d52a0dc admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:49.710611 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6f3e20d-f561-495f-abc1-b4d54d52a0dc admin admin] PATH_INFO: `/v3/projects/01ad67f60b6c459a979014fbeaf203e1` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:49.713001 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 999/2000] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:34:49 2022] GET /identity/v3/roles => generated 1420 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:34:49.713199 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6f3e20d-f561-495f-abc1-b4d54d52a0dc admin admin] RBAC: Authorizing `identity:delete_project(project_id=01ad67f60b6c459a979014fbeaf203e1)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:49.716283 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6f3e20d-f561-495f-abc1-b4d54d52a0dc admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:49.717211 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c826f2e8-0d26-4406-b2de-3e382d9d86ea None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:49.717513 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c826f2e8-0d26-4406-b2de-3e382d9d86ea None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:49.717766 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c826f2e8-0d26-4406-b2de-3e382d9d86ea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:49.752465 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-c826f2e8-0d26-4406-b2de-3e382d9d86ea None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:34:49.755852 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-c826f2e8-0d26-4406-b2de-3e382d9d86ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:49.781659 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1000/2001] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:34:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 68 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:34:49.783534 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c826f2e8-0d26-4406-b2de-3e382d9d86ea None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:49.801376 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c826f2e8-0d26-4406-b2de-3e382d9d86ea None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:49.811927 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deaa81a9-cfe3-4fdc-b053-871d49422ac1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:49.813181 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deaa81a9-cfe3-4fdc-b053-871d49422ac1 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:49.813766 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deaa81a9-cfe3-4fdc-b053-871d49422ac1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:49.814288 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deaa81a9-cfe3-4fdc-b053-871d49422ac1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:49.815267 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-deaa81a9-cfe3-4fdc-b053-871d49422ac1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:49.818216 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-deaa81a9-cfe3-4fdc-b053-871d49422ac1 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:49.825594 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-e6f3e20d-f561-495f-abc1-b4d54d52a0dc admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '01ad67f60b6c459a979014fbeaf203e1', 'request_id': 'req-e6f3e20d-f561-495f-abc1-b4d54d52a0dc'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:49.836525 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-e6f3e20d-f561-495f-abc1-b4d54d52a0dc admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '01ad67f60b6c459a979014fbeaf203e1', 'request_id': 'req-e6f3e20d-f561-495f-abc1-b4d54d52a0dc'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:49.845247 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-e6f3e20d-f561-495f-abc1-b4d54d52a0dc admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '01ad67f60b6c459a979014fbeaf203e1', 'request_id': 'req-e6f3e20d-f561-495f-abc1-b4d54d52a0dc'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:49.851709 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1001/2002] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:34:49 2022] GET /identity/v3/roles => generated 1420 bytes in 66 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:34:49.851709 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd703a6-1452-431e-9af7-d5789cc4271e None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:49.851709 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd703a6-1452-431e-9af7-d5789cc4271e None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:49.852071 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd703a6-1452-431e-9af7-d5789cc4271e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:49.871774 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-e6f3e20d-f561-495f-abc1-b4d54d52a0dc admin admin] The token cache is being invalidate because project 01ad67f60b6c459a979014fbeaf203e1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81434) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:34:49.872315 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-e6f3e20d-f561-495f-abc1-b4d54d52a0dc admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:49.873992 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-e6f3e20d-f561-495f-abc1-b4d54d52a0dc admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:49.876185 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1002/2003] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:34:49 2022] DELETE /identity/v3/projects/01ad67f60b6c459a979014fbeaf203e1 => generated 0 bytes in 198 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:34:49.877231 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6f3e20d-f561-495f-abc1-b4d54d52a0dc admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:49.882460 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-e6f3e20d-f561-495f-abc1-b4d54d52a0dc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:49.892579 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-2dd703a6-1452-431e-9af7-d5789cc4271e None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:34:49.894673 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-2dd703a6-1452-431e-9af7-d5789cc4271e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:50.015527 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1002/2004] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:34:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:34:50.015886 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6f3e20d-f561-495f-abc1-b4d54d52a0dc admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:50.018747 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dd703a6-1452-431e-9af7-d5789cc4271e None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:50.021063 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-2dd703a6-1452-431e-9af7-d5789cc4271e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:50.026670 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-318433f3-7205-44e9-8c0a-c31423d9ed09 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:50.027364 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-318433f3-7205-44e9-8c0a-c31423d9ed09 admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:50.027655 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-318433f3-7205-44e9-8c0a-c31423d9ed09 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:50.027955 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-318433f3-7205-44e9-8c0a-c31423d9ed09 admin admin] PATH_INFO: `/v3/projects/9b54c513066a4e8289eec3a708d82256/users/0782c69725e54da597c140a55ffd6e89/roles/02e242d5520740d4b77a5f4008c7eb57` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:50.041084 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dd703a6-1452-431e-9af7-d5789cc4271e None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:50.046767 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-843cbdd6-d72e-4689-acff-261faf942564 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:50.047553 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-843cbdd6-d72e-4689-acff-261faf942564 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:50.047837 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-843cbdd6-d72e-4689-acff-261faf942564 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:50.048312 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-843cbdd6-d72e-4689-acff-261faf942564 admin admin] PATH_INFO: `/v3/users/0a147ed4877a4e08b9ea10fa4098ed73` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:50.048420 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-318433f3-7205-44e9-8c0a-c31423d9ed09 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9b54c513066a4e8289eec3a708d82256, user_id=0782c69725e54da597c140a55ffd6e89, role_id=02e242d5520740d4b77a5f4008c7eb57)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:50.049397 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-843cbdd6-d72e-4689-acff-261faf942564 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0a147ed4877a4e08b9ea10fa4098ed73)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:50.051424 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-843cbdd6-d72e-4689-acff-261faf942564 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:50.058786 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-318433f3-7205-44e9-8c0a-c31423d9ed09 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:50.079863 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:34:50.099490 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1003/2005] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:34:49 2022] PUT /identity/v3/projects/9b54c513066a4e8289eec3a708d82256/users/0782c69725e54da597c140a55ffd6e89/roles/02e242d5520740d4b77a5f4008c7eb57 => generated 0 bytes in 223 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:34:50.101206 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-318433f3-7205-44e9-8c0a-c31423d9ed09 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:50.121902 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:34:50.147244 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-843cbdd6-d72e-4689-acff-261faf942564 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0a147ed4877a4e08b9ea10fa4098ed73'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:50.182455 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-843cbdd6-d72e-4689-acff-261faf942564 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0a147ed4877a4e08b9ea10fa4098ed73'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:50.183422 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-843cbdd6-d72e-4689-acff-261faf942564 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0a147ed4877a4e08b9ea10fa4098ed73'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:50.184667 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-843cbdd6-d72e-4689-acff-261faf942564 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0a147ed4877a4e08b9ea10fa4098ed73'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:50.199822 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-843cbdd6-d72e-4689-acff-261faf942564 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0a147ed4877a4e08b9ea10fa4098ed73'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:50.220459 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1003/2006] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:34:50 2022] DELETE /identity/v3/users/0a147ed4877a4e08b9ea10fa4098ed73 => generated 0 bytes in 204 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:34:50.223121 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2378a05e-d681-4543-b797-3eb589cdb8aa None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:50.223851 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-318433f3-7205-44e9-8c0a-c31423d9ed09 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:50.225760 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2378a05e-d681-4543-b797-3eb589cdb8aa None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:50.230258 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2378a05e-d681-4543-b797-3eb589cdb8aa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:50.288807 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-2378a05e-d681-4543-b797-3eb589cdb8aa None None] MFA Rules not processed for user `0782c69725e54da597c140a55ffd6e89`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:34:50.290899 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-2378a05e-d681-4543-b797-3eb589cdb8aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:50.357459 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-477642a8-6f82-4ee3-87be-64f231ab17b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:50.359422 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-477642a8-6f82-4ee3-87be-64f231ab17b7 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:50.359881 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-477642a8-6f82-4ee3-87be-64f231ab17b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:50.360178 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-477642a8-6f82-4ee3-87be-64f231ab17b7 admin admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:50.360860 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-477642a8-6f82-4ee3-87be-64f231ab17b7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:34:50.361322 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-477642a8-6f82-4ee3-87be-64f231ab17b7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:50.366223 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-477642a8-6f82-4ee3-87be-64f231ab17b7 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:50.385837 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1004/2007] 10.222.0.90 () {58 vars in 1153 bytes} [Mon Aug 29 21:34:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 285 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:34:50.387544 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-477642a8-6f82-4ee3-87be-64f231ab17b7 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:50.391798 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-477642a8-6f82-4ee3-87be-64f231ab17b7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:50.491934 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1004/2008] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:34:50 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 270 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:34:50.492990 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2378a05e-d681-4543-b797-3eb589cdb8aa None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:50.496317 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-2378a05e-d681-4543-b797-3eb589cdb8aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:50.537289 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2378a05e-d681-4543-b797-3eb589cdb8aa None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:50.550576 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-477642a8-6f82-4ee3-87be-64f231ab17b7 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:50.552690 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-155a4499-d529-48c6-a038-948d386c2072 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:50.553569 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-155a4499-d529-48c6-a038-948d386c2072 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:50.553890 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-155a4499-d529-48c6-a038-948d386c2072 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:50.554284 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-155a4499-d529-48c6-a038-948d386c2072 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:50.556276 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-155a4499-d529-48c6-a038-948d386c2072 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:50.559872 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8ae92e8-2617-44df-80d0-534c8f0e1367 tempest-VolumeQuotaClassesTest-1375012853 tempest-VolumeQuotaClassesTest-1375012853-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '572ed224ebbc4e558a4c7c2ce8201bd5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cb3bc5833ea94b3bacca8826b813b662', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:50.561778 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8ae92e8-2617-44df-80d0-534c8f0e1367 tempest-VolumeQuotaClassesTest-1375012853 tempest-VolumeQuotaClassesTest-1375012853-project-admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:50.562361 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-155a4499-d529-48c6-a038-948d386c2072 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:50.563599 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8ae92e8-2617-44df-80d0-534c8f0e1367 tempest-VolumeQuotaClassesTest-1375012853 tempest-VolumeQuotaClassesTest-1375012853-project-admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:50.564460 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8ae92e8-2617-44df-80d0-534c8f0e1367 tempest-VolumeQuotaClassesTest-1375012853 tempest-VolumeQuotaClassesTest-1375012853-project-admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:50.565688 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8ae92e8-2617-44df-80d0-534c8f0e1367 tempest-VolumeQuotaClassesTest-1375012853 tempest-VolumeQuotaClassesTest-1375012853-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:34:50.566414 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8ae92e8-2617-44df-80d0-534c8f0e1367 tempest-VolumeQuotaClassesTest-1375012853 tempest-VolumeQuotaClassesTest-1375012853-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:50.570055 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8ae92e8-2617-44df-80d0-534c8f0e1367 tempest-VolumeQuotaClassesTest-1375012853 tempest-VolumeQuotaClassesTest-1375012853-project-admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:50.593878 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1005/2009] 10.222.0.90 () {58 vars in 1153 bytes} [Mon Aug 29 21:34:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:34:50.595102 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8ae92e8-2617-44df-80d0-534c8f0e1367 tempest-VolumeQuotaClassesTest-1375012853 tempest-VolumeQuotaClassesTest-1375012853-project-admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:50.599140 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-b8ae92e8-2617-44df-80d0-534c8f0e1367 tempest-VolumeQuotaClassesTest-1375012853 tempest-VolumeQuotaClassesTest-1375012853-project-admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:50.627466 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8ae92e8-2617-44df-80d0-534c8f0e1367 tempest-VolumeQuotaClassesTest-1375012853 tempest-VolumeQuotaClassesTest-1375012853-project-admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:50.630398 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1005/2010] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:34:50 2022] POST /identity/v3/projects => generated 381 bytes in 138 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:34:50.631844 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-155a4499-d529-48c6-a038-948d386c2072 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:50.694024 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:34:50.729758 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0caaad0-3f84-4a49-b674-bc2474b73b52 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:50.730577 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0caaad0-3f84-4a49-b674-bc2474b73b52 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:50.730871 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0caaad0-3f84-4a49-b674-bc2474b73b52 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:50.731146 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0caaad0-3f84-4a49-b674-bc2474b73b52 admin admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:50.731818 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0caaad0-3f84-4a49-b674-bc2474b73b52 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:50.733886 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0caaad0-3f84-4a49-b674-bc2474b73b52 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:50.799805 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-155a4499-d529-48c6-a038-948d386c2072 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:50.810102 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e34c8615-1bcf-4bc8-8a94-6e8d15fdb569 tempest-VolumeQuotaClassesTest-1375012853 tempest-VolumeQuotaClassesTest-1375012853-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '572ed224ebbc4e558a4c7c2ce8201bd5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cb3bc5833ea94b3bacca8826b813b662', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:50.810833 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e34c8615-1bcf-4bc8-8a94-6e8d15fdb569 tempest-VolumeQuotaClassesTest-1375012853 tempest-VolumeQuotaClassesTest-1375012853-project-admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:50.811073 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e34c8615-1bcf-4bc8-8a94-6e8d15fdb569 tempest-VolumeQuotaClassesTest-1375012853 tempest-VolumeQuotaClassesTest-1375012853-project-admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:50.811485 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e34c8615-1bcf-4bc8-8a94-6e8d15fdb569 tempest-VolumeQuotaClassesTest-1375012853 tempest-VolumeQuotaClassesTest-1375012853-project-admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:50.812944 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e34c8615-1bcf-4bc8-8a94-6e8d15fdb569 tempest-VolumeQuotaClassesTest-1375012853 tempest-VolumeQuotaClassesTest-1375012853-project-admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:50.818304 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1006/2011] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:34:50 2022] POST /identity/v3/projects => generated 367 bytes in 219 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:34:50.818304 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0caaad0-3f84-4a49-b674-bc2474b73b52 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:50.818576 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e34c8615-1bcf-4bc8-8a94-6e8d15fdb569 tempest-VolumeQuotaClassesTest-1375012853 tempest-VolumeQuotaClassesTest-1375012853-project-admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:50.852044 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=92,keystone:DELETE=5,keystone:INSERT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:34:50.868595 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1006/2012] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:34:50 2022] POST /identity/v3/projects => generated 354 bytes in 234 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:34:50.868595 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e34c8615-1bcf-4bc8-8a94-6e8d15fdb569 tempest-VolumeQuotaClassesTest-1375012853 tempest-VolumeQuotaClassesTest-1375012853-project-admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:50.874565 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-e34c8615-1bcf-4bc8-8a94-6e8d15fdb569 tempest-VolumeQuotaClassesTest-1375012853 tempest-VolumeQuotaClassesTest-1375012853-project-admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:50.924105 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0caaad0-3f84-4a49-b674-bc2474b73b52 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:50.992097 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e34c8615-1bcf-4bc8-8a94-6e8d15fdb569 tempest-VolumeQuotaClassesTest-1375012853 tempest-VolumeQuotaClassesTest-1375012853-project-admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:51.001564 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2888ef6c-2ff4-46bc-a298-3c3d8f40bb99 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:51.002252 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2888ef6c-2ff4-46bc-a298-3c3d8f40bb99 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:51.002534 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2888ef6c-2ff4-46bc-a298-3c3d8f40bb99 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:51.002806 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2888ef6c-2ff4-46bc-a298-3c3d8f40bb99 admin admin] PATH_INFO: `/v3/projects/b8d69910a0ca48a5b09b918cb326244a` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:51.004960 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2888ef6c-2ff4-46bc-a298-3c3d8f40bb99 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b8d69910a0ca48a5b09b918cb326244a)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:51.009709 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2888ef6c-2ff4-46bc-a298-3c3d8f40bb99 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:51.049862 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03fa58d8-20ef-4d19-bdd9-94e96e180342 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:51.050863 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03fa58d8-20ef-4d19-bdd9-94e96e180342 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:51.051628 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03fa58d8-20ef-4d19-bdd9-94e96e180342 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:51.066591 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03fa58d8-20ef-4d19-bdd9-94e96e180342 admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:51.067329 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03fa58d8-20ef-4d19-bdd9-94e96e180342 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:51.069497 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03fa58d8-20ef-4d19-bdd9-94e96e180342 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:51.116272 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1007/2013] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:34:50 2022] POST /identity/v3/users => generated 291 bytes in 302 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:34:51.117127 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03fa58d8-20ef-4d19-bdd9-94e96e180342 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:51.123041 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-2888ef6c-2ff4-46bc-a298-3c3d8f40bb99 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b8d69910a0ca48a5b09b918cb326244a', 'request_id': 'req-2888ef6c-2ff4-46bc-a298-3c3d8f40bb99'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:51.130954 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03fa58d8-20ef-4d19-bdd9-94e96e180342 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:51.135221 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-2888ef6c-2ff4-46bc-a298-3c3d8f40bb99 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b8d69910a0ca48a5b09b918cb326244a', 'request_id': 'req-2888ef6c-2ff4-46bc-a298-3c3d8f40bb99'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:51.149691 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-2888ef6c-2ff4-46bc-a298-3c3d8f40bb99 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b8d69910a0ca48a5b09b918cb326244a', 'request_id': 'req-2888ef6c-2ff4-46bc-a298-3c3d8f40bb99'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:51.170156 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-2888ef6c-2ff4-46bc-a298-3c3d8f40bb99 admin admin] The token cache is being invalidate because project b8d69910a0ca48a5b09b918cb326244a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:34:51.170511 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-2888ef6c-2ff4-46bc-a298-3c3d8f40bb99 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:51.171404 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-2888ef6c-2ff4-46bc-a298-3c3d8f40bb99 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:51.174040 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1007/2014] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:34:50 2022] DELETE /identity/v3/projects/b8d69910a0ca48a5b09b918cb326244a => generated 0 bytes in 308 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:34:51.175833 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2888ef6c-2ff4-46bc-a298-3c3d8f40bb99 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:51.181581 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-2888ef6c-2ff4-46bc-a298-3c3d8f40bb99 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:51.250060 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:34:51.259114 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae9beb3f-22a6-4c76-bc37-7fd92313ddc5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:51.259934 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae9beb3f-22a6-4c76-bc37-7fd92313ddc5 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:51.260656 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae9beb3f-22a6-4c76-bc37-7fd92313ddc5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:51.261015 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae9beb3f-22a6-4c76-bc37-7fd92313ddc5 admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:51.261975 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae9beb3f-22a6-4c76-bc37-7fd92313ddc5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:51.265517 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae9beb3f-22a6-4c76-bc37-7fd92313ddc5 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:51.290686 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1008/2015] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:34:51 2022] POST /identity/v3/users => generated 290 bytes in 174 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:34:51.294937 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae9beb3f-22a6-4c76-bc37-7fd92313ddc5 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:51.296881 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-ae9beb3f-22a6-4c76-bc37-7fd92313ddc5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:51.309675 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae9beb3f-22a6-4c76-bc37-7fd92313ddc5 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:51.310473 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2888ef6c-2ff4-46bc-a298-3c3d8f40bb99 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:51.314860 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6fa35d3-c8ab-490f-9560-abe503ab033c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:51.315556 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6fa35d3-c8ab-490f-9560-abe503ab033c admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:51.316333 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6fa35d3-c8ab-490f-9560-abe503ab033c admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:51.316682 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6fa35d3-c8ab-490f-9560-abe503ab033c admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:51.317285 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aca262f-3b61-49cb-87f2-00f22875a4d7 tempest-VolumeQuotaClassesTest-1375012853 tempest-VolumeQuotaClassesTest-1375012853-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '572ed224ebbc4e558a4c7c2ce8201bd5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cb3bc5833ea94b3bacca8826b813b662', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:51.318167 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aca262f-3b61-49cb-87f2-00f22875a4d7 tempest-VolumeQuotaClassesTest-1375012853 tempest-VolumeQuotaClassesTest-1375012853-project-admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:51.318332 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6fa35d3-c8ab-490f-9560-abe503ab033c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:51.318462 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aca262f-3b61-49cb-87f2-00f22875a4d7 tempest-VolumeQuotaClassesTest-1375012853 tempest-VolumeQuotaClassesTest-1375012853-project-admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:51.318885 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aca262f-3b61-49cb-87f2-00f22875a4d7 tempest-VolumeQuotaClassesTest-1375012853 tempest-VolumeQuotaClassesTest-1375012853-project-admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:51.319605 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aca262f-3b61-49cb-87f2-00f22875a4d7 tempest-VolumeQuotaClassesTest-1375012853 tempest-VolumeQuotaClassesTest-1375012853-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:34:51.319985 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aca262f-3b61-49cb-87f2-00f22875a4d7 tempest-VolumeQuotaClassesTest-1375012853 tempest-VolumeQuotaClassesTest-1375012853-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:51.320237 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6fa35d3-c8ab-490f-9560-abe503ab033c admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:51.321694 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aca262f-3b61-49cb-87f2-00f22875a4d7 tempest-VolumeQuotaClassesTest-1375012853 tempest-VolumeQuotaClassesTest-1375012853-project-admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:51.335437 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1008/2016] 10.222.0.90 () {58 vars in 1153 bytes} [Mon Aug 29 21:34:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:34:51.336791 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a97d3e-7d5b-424f-b334-168319147663 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:51.337006 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a97d3e-7d5b-424f-b334-168319147663 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:51.337546 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a97d3e-7d5b-424f-b334-168319147663 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:51.342036 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1009/2017] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:34:51 2022] GET /identity/v3/roles => generated 1420 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:34:51.342036 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6fa35d3-c8ab-490f-9560-abe503ab033c admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:51.343505 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-c6fa35d3-c8ab-490f-9560-abe503ab033c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:51.359148 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6fa35d3-c8ab-490f-9560-abe503ab033c admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:51.362158 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-d1a97d3e-7d5b-424f-b334-168319147663 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:34:51.363584 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-d1a97d3e-7d5b-424f-b334-168319147663 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:51.368145 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d7d9e43-4f45-4b10-91e3-d48265d1cc12 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:51.370206 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d7d9e43-4f45-4b10-91e3-d48265d1cc12 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:51.370922 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d7d9e43-4f45-4b10-91e3-d48265d1cc12 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:51.371522 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d7d9e43-4f45-4b10-91e3-d48265d1cc12 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:51.372424 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d7d9e43-4f45-4b10-91e3-d48265d1cc12 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:51.373838 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1009/2018] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:34:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:34:51.374785 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d7d9e43-4f45-4b10-91e3-d48265d1cc12 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:51.374900 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1a97d3e-7d5b-424f-b334-168319147663 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:51.386019 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1a97d3e-7d5b-424f-b334-168319147663 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:51.391472 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a13e91f4-5a44-4d0a-9e3b-03f97106f00c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:51.392253 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a13e91f4-5a44-4d0a-9e3b-03f97106f00c admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:51.392945 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a13e91f4-5a44-4d0a-9e3b-03f97106f00c admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:51.393125 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1010/2019] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:34:51 2022] GET /identity/v3/roles => generated 1420 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:34:51.393289 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a13e91f4-5a44-4d0a-9e3b-03f97106f00c admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:51.393938 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a13e91f4-5a44-4d0a-9e3b-03f97106f00c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:51.396684 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a13e91f4-5a44-4d0a-9e3b-03f97106f00c admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:51.400931 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2276fc57-d581-4469-a32c-3be1dab983ff None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:51.401229 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2276fc57-d581-4469-a32c-3be1dab983ff None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:51.401639 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2276fc57-d581-4469-a32c-3be1dab983ff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:51.416244 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1010/2020] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:34:51 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:34:51.416646 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a13e91f4-5a44-4d0a-9e3b-03f97106f00c admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:51.431259 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-2276fc57-d581-4469-a32c-3be1dab983ff None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:34:51.432777 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-2276fc57-d581-4469-a32c-3be1dab983ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:51.434890 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a13e91f4-5a44-4d0a-9e3b-03f97106f00c admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:51.440583 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a99a53d0-00b9-4d9f-b911-ea448e6d0ed5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:51.441265 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a99a53d0-00b9-4d9f-b911-ea448e6d0ed5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:51.445271 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1011/2021] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:34:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 48 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:34:51.445429 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a99a53d0-00b9-4d9f-b911-ea448e6d0ed5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:51.445429 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a99a53d0-00b9-4d9f-b911-ea448e6d0ed5 admin admin] PATH_INFO: `/v3/projects/dea618c99b294c0ba9e8484dcd731e73/users/22c1b225b1044334a2de2761d584f048/roles/a6e562cdc96c4be691d638990f8403fd` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:51.445668 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2276fc57-d581-4469-a32c-3be1dab983ff None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:51.450460 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:34:51.463260 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2276fc57-d581-4469-a32c-3be1dab983ff None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:51.470212 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a99a53d0-00b9-4d9f-b911-ea448e6d0ed5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=dea618c99b294c0ba9e8484dcd731e73, user_id=22c1b225b1044334a2de2761d584f048, role_id=a6e562cdc96c4be691d638990f8403fd)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:51.471795 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-528287b6-09dc-43d3-a0ad-ba62697fadc2 tempest-VolumeQuotaClassesTest-1375012853 tempest-VolumeQuotaClassesTest-1375012853-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '572ed224ebbc4e558a4c7c2ce8201bd5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cb3bc5833ea94b3bacca8826b813b662', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:51.472666 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-528287b6-09dc-43d3-a0ad-ba62697fadc2 tempest-VolumeQuotaClassesTest-1375012853 tempest-VolumeQuotaClassesTest-1375012853-project-admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:51.472951 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-528287b6-09dc-43d3-a0ad-ba62697fadc2 tempest-VolumeQuotaClassesTest-1375012853 tempest-VolumeQuotaClassesTest-1375012853-project-admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:51.473562 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-528287b6-09dc-43d3-a0ad-ba62697fadc2 tempest-VolumeQuotaClassesTest-1375012853 tempest-VolumeQuotaClassesTest-1375012853-project-admin] PATH_INFO: `/v3/projects/fbbe5f12e4ea4dd7b6bbf4e267ad0c85` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:51.473695 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a99a53d0-00b9-4d9f-b911-ea448e6d0ed5 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:51.475104 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-528287b6-09dc-43d3-a0ad-ba62697fadc2 tempest-VolumeQuotaClassesTest-1375012853 tempest-VolumeQuotaClassesTest-1375012853-project-admin] RBAC: Authorizing `identity:delete_project(project_id=fbbe5f12e4ea4dd7b6bbf4e267ad0c85)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:51.477974 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-528287b6-09dc-43d3-a0ad-ba62697fadc2 tempest-VolumeQuotaClassesTest-1375012853 tempest-VolumeQuotaClassesTest-1375012853-project-admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:51.491313 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1011/2022] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:34:51 2022] PUT /identity/v3/projects/dea618c99b294c0ba9e8484dcd731e73/users/22c1b225b1044334a2de2761d584f048/roles/a6e562cdc96c4be691d638990f8403fd => generated 0 bytes in 75 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:34:51.492821 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a99a53d0-00b9-4d9f-b911-ea448e6d0ed5 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:51.570361 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-528287b6-09dc-43d3-a0ad-ba62697fadc2 tempest-VolumeQuotaClassesTest-1375012853 tempest-VolumeQuotaClassesTest-1375012853-project-admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fbbe5f12e4ea4dd7b6bbf4e267ad0c85', 'request_id': 'req-528287b6-09dc-43d3-a0ad-ba62697fadc2'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:51.582050 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-528287b6-09dc-43d3-a0ad-ba62697fadc2 tempest-VolumeQuotaClassesTest-1375012853 tempest-VolumeQuotaClassesTest-1375012853-project-admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fbbe5f12e4ea4dd7b6bbf4e267ad0c85', 'request_id': 'req-528287b6-09dc-43d3-a0ad-ba62697fadc2'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:51.592630 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-528287b6-09dc-43d3-a0ad-ba62697fadc2 tempest-VolumeQuotaClassesTest-1375012853 tempest-VolumeQuotaClassesTest-1375012853-project-admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fbbe5f12e4ea4dd7b6bbf4e267ad0c85', 'request_id': 'req-528287b6-09dc-43d3-a0ad-ba62697fadc2'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:51.618092 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a99a53d0-00b9-4d9f-b911-ea448e6d0ed5 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:51.621334 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-528287b6-09dc-43d3-a0ad-ba62697fadc2 tempest-VolumeQuotaClassesTest-1375012853 tempest-VolumeQuotaClassesTest-1375012853-project-admin] The token cache is being invalidate because project fbbe5f12e4ea4dd7b6bbf4e267ad0c85 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81434) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:34:51.622129 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-528287b6-09dc-43d3-a0ad-ba62697fadc2 tempest-VolumeQuotaClassesTest-1375012853 tempest-VolumeQuotaClassesTest-1375012853-project-admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:51.623330 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-528287b6-09dc-43d3-a0ad-ba62697fadc2 tempest-VolumeQuotaClassesTest-1375012853 tempest-VolumeQuotaClassesTest-1375012853-project-admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:51.626542 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1012/2023] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:34:51 2022] DELETE /identity/v3/projects/fbbe5f12e4ea4dd7b6bbf4e267ad0c85 => generated 0 bytes in 184 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:34:51.628120 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-528287b6-09dc-43d3-a0ad-ba62697fadc2 tempest-VolumeQuotaClassesTest-1375012853 tempest-VolumeQuotaClassesTest-1375012853-project-admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:51.631065 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-528287b6-09dc-43d3-a0ad-ba62697fadc2 tempest-VolumeQuotaClassesTest-1375012853 tempest-VolumeQuotaClassesTest-1375012853-project-admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:51.744487 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-528287b6-09dc-43d3-a0ad-ba62697fadc2 tempest-VolumeQuotaClassesTest-1375012853 tempest-VolumeQuotaClassesTest-1375012853-project-admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:51.749243 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03a0e37b-2cc7-4ed5-b5fe-0ab93af4dac3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:51.750487 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03a0e37b-2cc7-4ed5-b5fe-0ab93af4dac3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:51.750968 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03a0e37b-2cc7-4ed5-b5fe-0ab93af4dac3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:51.751276 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03a0e37b-2cc7-4ed5-b5fe-0ab93af4dac3 admin admin] PATH_INFO: `/v3/projects/604e07baa03b4cb5a0e2c94bfd59993d/users/e818fd39b54f41078d3c00bdb547ff9f/roles/02e242d5520740d4b77a5f4008c7eb57` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:51.753281 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8fdc9a2-529d-4821-b11a-7200841300d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:51.754371 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8fdc9a2-529d-4821-b11a-7200841300d8 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:51.754790 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8fdc9a2-529d-4821-b11a-7200841300d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:51.755205 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8fdc9a2-529d-4821-b11a-7200841300d8 admin admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:51.756361 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8fdc9a2-529d-4821-b11a-7200841300d8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:34:51.756877 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8fdc9a2-529d-4821-b11a-7200841300d8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:51.760132 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8fdc9a2-529d-4821-b11a-7200841300d8 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:51.784116 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03a0e37b-2cc7-4ed5-b5fe-0ab93af4dac3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=604e07baa03b4cb5a0e2c94bfd59993d, user_id=e818fd39b54f41078d3c00bdb547ff9f, role_id=02e242d5520740d4b77a5f4008c7eb57)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:51.789077 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03a0e37b-2cc7-4ed5-b5fe-0ab93af4dac3 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:51.792723 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1013/2024] 10.222.0.90 () {58 vars in 1153 bytes} [Mon Aug 29 21:34:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:34:51.799256 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8fdc9a2-529d-4821-b11a-7200841300d8 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:51.803180 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-a8fdc9a2-529d-4821-b11a-7200841300d8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:51.816405 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1012/2025] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:34:51 2022] PUT /identity/v3/projects/604e07baa03b4cb5a0e2c94bfd59993d/users/e818fd39b54f41078d3c00bdb547ff9f/roles/02e242d5520740d4b77a5f4008c7eb57 => generated 0 bytes in 325 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:34:51.818090 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03a0e37b-2cc7-4ed5-b5fe-0ab93af4dac3 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:51.842984 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8fdc9a2-529d-4821-b11a-7200841300d8 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:51.964739 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=89,keystone:INSERT=4,keystone:DELETE=7 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:34:52.003275 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03a0e37b-2cc7-4ed5-b5fe-0ab93af4dac3 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:52.011998 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be4a30bf-e4b0-4561-b888-e94b78ae70b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:52.013392 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be4a30bf-e4b0-4561-b888-e94b78ae70b2 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:52.013894 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be4a30bf-e4b0-4561-b888-e94b78ae70b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:52.014435 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be4a30bf-e4b0-4561-b888-e94b78ae70b2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:52.015656 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be4a30bf-e4b0-4561-b888-e94b78ae70b2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:52.018967 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be4a30bf-e4b0-4561-b888-e94b78ae70b2 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:52.061290 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf2fdeaf-dbfa-409a-b784-cacd6ad51d2b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:52.062696 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf2fdeaf-dbfa-409a-b784-cacd6ad51d2b admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:52.063289 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf2fdeaf-dbfa-409a-b784-cacd6ad51d2b admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:52.063807 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf2fdeaf-dbfa-409a-b784-cacd6ad51d2b admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:52.064402 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:34:52.064999 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf2fdeaf-dbfa-409a-b784-cacd6ad51d2b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:52.069527 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf2fdeaf-dbfa-409a-b784-cacd6ad51d2b admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:52.080296 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1013/2026] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:34:51 2022] POST /identity/v3/projects => generated 375 bytes in 263 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:34:52.083720 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf748a63-7670-4640-88cd-3cf9def5a032 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:52.084613 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf748a63-7670-4640-88cd-3cf9def5a032 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:52.085393 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf748a63-7670-4640-88cd-3cf9def5a032 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:52.118850 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1014/2027] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:34:51 2022] GET /identity/v3/roles => generated 1420 bytes in 317 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:34:52.118850 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf2fdeaf-dbfa-409a-b784-cacd6ad51d2b admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:52.132415 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf2fdeaf-dbfa-409a-b784-cacd6ad51d2b admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:52.147427 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-cf748a63-7670-4640-88cd-3cf9def5a032 None None] MFA Rules not processed for user `e818fd39b54f41078d3c00bdb547ff9f`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:34:52.150949 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-cf748a63-7670-4640-88cd-3cf9def5a032 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:52.219595 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5af2a54c-b2e7-431b-b15e-18c473ee8268 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:52.220236 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5af2a54c-b2e7-431b-b15e-18c473ee8268 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:52.220671 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5af2a54c-b2e7-431b-b15e-18c473ee8268 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:52.220980 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5af2a54c-b2e7-431b-b15e-18c473ee8268 admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:52.221706 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5af2a54c-b2e7-431b-b15e-18c473ee8268 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:52.224275 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5af2a54c-b2e7-431b-b15e-18c473ee8268 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:52.256320 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1015/2028] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:34:52 2022] POST /identity/v3/users => generated 288 bytes in 140 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:34:52.258039 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5af2a54c-b2e7-431b-b15e-18c473ee8268 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:52.258975 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1014/2029] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:34:52 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:34:52.261331 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf748a63-7670-4640-88cd-3cf9def5a032 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:52.271063 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-cf748a63-7670-4640-88cd-3cf9def5a032 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:52.284696 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5af2a54c-b2e7-431b-b15e-18c473ee8268 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:52.294707 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94ac0586-ea0a-425a-b8b4-674a6422dcb9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:52.296008 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94ac0586-ea0a-425a-b8b4-674a6422dcb9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:52.296729 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94ac0586-ea0a-425a-b8b4-674a6422dcb9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:52.297205 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94ac0586-ea0a-425a-b8b4-674a6422dcb9 admin admin] PATH_INFO: `/v3/projects/dea618c99b294c0ba9e8484dcd731e73/users/22c1b225b1044334a2de2761d584f048/roles/a6e562cdc96c4be691d638990f8403fd` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:52.301833 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94ac0586-ea0a-425a-b8b4-674a6422dcb9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=dea618c99b294c0ba9e8484dcd731e73, user_id=22c1b225b1044334a2de2761d584f048, role_id=a6e562cdc96c4be691d638990f8403fd)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:52.302707 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf748a63-7670-4640-88cd-3cf9def5a032 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:52.307465 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94ac0586-ea0a-425a-b8b4-674a6422dcb9 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:52.307737 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-289e3632-da09-4bcb-8081-2465f6b28376 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:52.308330 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-289e3632-da09-4bcb-8081-2465f6b28376 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:52.308613 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-289e3632-da09-4bcb-8081-2465f6b28376 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:52.308881 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-289e3632-da09-4bcb-8081-2465f6b28376 admin admin] PATH_INFO: `/v3/users/878148ec3d3a4ecba9f7dfa55e1bc27e` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:52.310245 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-289e3632-da09-4bcb-8081-2465f6b28376 admin admin] RBAC: Authorizing `identity:delete_user(user_id=878148ec3d3a4ecba9f7dfa55e1bc27e)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:52.312185 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-289e3632-da09-4bcb-8081-2465f6b28376 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:52.333345 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1016/2030] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:34:52 2022] PUT /identity/v3/projects/dea618c99b294c0ba9e8484dcd731e73/users/22c1b225b1044334a2de2761d584f048/roles/a6e562cdc96c4be691d638990f8403fd => generated 0 bytes in 76 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:34:52.334545 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94ac0586-ea0a-425a-b8b4-674a6422dcb9 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:52.416825 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-289e3632-da09-4bcb-8081-2465f6b28376 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '878148ec3d3a4ecba9f7dfa55e1bc27e'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:52.438173 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-289e3632-da09-4bcb-8081-2465f6b28376 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '878148ec3d3a4ecba9f7dfa55e1bc27e'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:52.439341 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-289e3632-da09-4bcb-8081-2465f6b28376 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '878148ec3d3a4ecba9f7dfa55e1bc27e'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:52.440116 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-289e3632-da09-4bcb-8081-2465f6b28376 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '878148ec3d3a4ecba9f7dfa55e1bc27e'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:52.450287 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-289e3632-da09-4bcb-8081-2465f6b28376 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '878148ec3d3a4ecba9f7dfa55e1bc27e'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:52.482374 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1015/2031] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:34:52 2022] DELETE /identity/v3/users/878148ec3d3a4ecba9f7dfa55e1bc27e => generated 0 bytes in 222 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:34:52.483735 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-289e3632-da09-4bcb-8081-2465f6b28376 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:52.486422 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-289e3632-da09-4bcb-8081-2465f6b28376 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:52.515201 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94ac0586-ea0a-425a-b8b4-674a6422dcb9 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:52.605768 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-289e3632-da09-4bcb-8081-2465f6b28376 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:52.614635 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55f352f2-667e-4ddc-8189-6bb0550ddebd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:52.615466 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55f352f2-667e-4ddc-8189-6bb0550ddebd admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:52.615930 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55f352f2-667e-4ddc-8189-6bb0550ddebd admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:52.616401 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55f352f2-667e-4ddc-8189-6bb0550ddebd admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:52.617593 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55f352f2-667e-4ddc-8189-6bb0550ddebd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:52.621277 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55f352f2-667e-4ddc-8189-6bb0550ddebd admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:52.624200 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:34:52.626780 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:34:52.642732 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-907c96ab-5580-4b99-bb35-4765a1b559dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:52.643874 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907c96ab-5580-4b99-bb35-4765a1b559dd admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:52.644440 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907c96ab-5580-4b99-bb35-4765a1b559dd admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:52.646443 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907c96ab-5580-4b99-bb35-4765a1b559dd admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:52.647815 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-907c96ab-5580-4b99-bb35-4765a1b559dd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:52.651709 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-907c96ab-5580-4b99-bb35-4765a1b559dd admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:52.668759 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1017/2032] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:34:52 2022] GET /identity/v3/roles => generated 1420 bytes in 335 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:34:52.670415 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-409144c6-48fc-46ea-b8c1-0cf0db03b77b None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:52.670696 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-409144c6-48fc-46ea-b8c1-0cf0db03b77b None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:52.671024 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-409144c6-48fc-46ea-b8c1-0cf0db03b77b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:52.675433 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1016/2033] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:34:52 2022] POST /identity/v3/projects => generated 385 bytes in 193 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:34:52.678278 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a866d1dc-3142-40e1-a312-357d428ccff9 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:52.678764 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a866d1dc-3142-40e1-a312-357d428ccff9 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:52.679245 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a866d1dc-3142-40e1-a312-357d428ccff9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:52.704373 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-a866d1dc-3142-40e1-a312-357d428ccff9 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:34:52.706395 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-a866d1dc-3142-40e1-a312-357d428ccff9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:52.742966 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-409144c6-48fc-46ea-b8c1-0cf0db03b77b None None] MFA Rules not processed for user `22c1b225b1044334a2de2761d584f048`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:34:52.745301 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-409144c6-48fc-46ea-b8c1-0cf0db03b77b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:52.878376 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1017/2034] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:34:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 202 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:34:52.883702 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1018/2035] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:34:52 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 214 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:34:52.884791 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-409144c6-48fc-46ea-b8c1-0cf0db03b77b None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:52.895177 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a866d1dc-3142-40e1-a312-357d428ccff9 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:52.898297 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-a866d1dc-3142-40e1-a312-357d428ccff9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:52.902339 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-409144c6-48fc-46ea-b8c1-0cf0db03b77b None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:52.913103 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a0ba81c-29d3-4b52-a751-73245cc83610 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:52.913890 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a0ba81c-29d3-4b52-a751-73245cc83610 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:52.914843 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a0ba81c-29d3-4b52-a751-73245cc83610 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:52.915191 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a0ba81c-29d3-4b52-a751-73245cc83610 admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:52.915958 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a0ba81c-29d3-4b52-a751-73245cc83610 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:52.919454 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a0ba81c-29d3-4b52-a751-73245cc83610 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:52.937899 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a866d1dc-3142-40e1-a312-357d428ccff9 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:52.948846 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f916e11-1067-42f6-b16f-6f567438c9fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:52.950310 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f916e11-1067-42f6-b16f-6f567438c9fa admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:52.950795 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f916e11-1067-42f6-b16f-6f567438c9fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:52.951489 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f916e11-1067-42f6-b16f-6f567438c9fa admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:52.952627 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f916e11-1067-42f6-b16f-6f567438c9fa admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:52.955727 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f916e11-1067-42f6-b16f-6f567438c9fa admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:52.956185 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1019/2036] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:34:52 2022] POST /identity/v3/users => generated 293 bytes in 72 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:34:52.957214 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a0ba81c-29d3-4b52-a751-73245cc83610 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:52.959594 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-5a0ba81c-29d3-4b52-a751-73245cc83610 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:52.991721 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1018/2037] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:34:52 2022] GET /identity/v3/roles => generated 1420 bytes in 113 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:34:52.992865 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f916e11-1067-42f6-b16f-6f567438c9fa admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:52.994761 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-5f916e11-1067-42f6-b16f-6f567438c9fa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:53.098569 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a0ba81c-29d3-4b52-a751-73245cc83610 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:53.107518 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-054ce29d-fa70-4e7e-bf57-bdb8020c2712 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:53.108673 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-054ce29d-fa70-4e7e-bf57-bdb8020c2712 service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:53.109441 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-054ce29d-fa70-4e7e-bf57-bdb8020c2712 service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:53.110044 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-054ce29d-fa70-4e7e-bf57-bdb8020c2712 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:53.125042 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f916e11-1067-42f6-b16f-6f567438c9fa admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:53.132613 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-054ce29d-fa70-4e7e-bf57-bdb8020c2712 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:53.134658 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-054ce29d-fa70-4e7e-bf57-bdb8020c2712 service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:53.140587 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d92f5f9d-0c3f-421c-a6c0-0e069728569a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dc226f7ddcb49ecacb51ce57fdfc11b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:53.142370 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d92f5f9d-0c3f-421c-a6c0-0e069728569a service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:53.142670 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d92f5f9d-0c3f-421c-a6c0-0e069728569a service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:53.142949 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d92f5f9d-0c3f-421c-a6c0-0e069728569a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:53.163642 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1020/2038] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:34:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 207 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:34:53.165332 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-054ce29d-fa70-4e7e-bf57-bdb8020c2712 service neutron] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:53.174752 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d92f5f9d-0c3f-421c-a6c0-0e069728569a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:53.178069 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d92f5f9d-0c3f-421c-a6c0-0e069728569a service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:53.189260 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-054ce29d-fa70-4e7e-bf57-bdb8020c2712 service neutron] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:53.198707 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f469da64-fce2-49d4-9beb-84310ec68470 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:53.199699 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f469da64-fce2-49d4-9beb-84310ec68470 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:53.200153 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f469da64-fce2-49d4-9beb-84310ec68470 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:53.200779 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f469da64-fce2-49d4-9beb-84310ec68470 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:53.202004 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f469da64-fce2-49d4-9beb-84310ec68470 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:53.205021 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f469da64-fce2-49d4-9beb-84310ec68470 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:53.218102 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1019/2039] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:34:52 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 225 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:34:53.220944 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d92f5f9d-0c3f-421c-a6c0-0e069728569a service cinder] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:53.233836 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1021/2040] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:34:53 2022] GET /identity/v3/roles => generated 1420 bytes in 69 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:34:53.235787 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f469da64-fce2-49d4-9beb-84310ec68470 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:53.247525 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d92f5f9d-0c3f-421c-a6c0-0e069728569a service cinder] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:53.254393 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f469da64-fce2-49d4-9beb-84310ec68470 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:53.258716 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aeaa687-c5ab-4385-a489-65b033853226 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:53.260222 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2dfe44e-e6f0-4091-b707-7bd6df123999 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:53.261016 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2dfe44e-e6f0-4091-b707-7bd6df123999 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:53.261168 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2dfe44e-e6f0-4091-b707-7bd6df123999 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:53.262167 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2dfe44e-e6f0-4091-b707-7bd6df123999 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:53.262167 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2dfe44e-e6f0-4091-b707-7bd6df123999 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:53.263464 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aeaa687-c5ab-4385-a489-65b033853226 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:53.264726 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2dfe44e-e6f0-4091-b707-7bd6df123999 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:53.265540 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aeaa687-c5ab-4385-a489-65b033853226 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:53.266559 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aeaa687-c5ab-4385-a489-65b033853226 admin admin] PATH_INFO: `/v3/projects/cac54fe0108d448abb161656c9c50bdd/users/e9e14e1e61c64e3aac538de88162d183/roles/02e242d5520740d4b77a5f4008c7eb57` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:53.283067 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1022/2041] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:34:53 2022] GET /identity/v3/roles => generated 1420 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:34:53.284038 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2dfe44e-e6f0-4091-b707-7bd6df123999 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:53.295119 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aeaa687-c5ab-4385-a489-65b033853226 admin admin] RBAC: Authorizing `identity:create_grant(project_id=cac54fe0108d448abb161656c9c50bdd, user_id=e9e14e1e61c64e3aac538de88162d183, role_id=02e242d5520740d4b77a5f4008c7eb57)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:53.295843 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2dfe44e-e6f0-4091-b707-7bd6df123999 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:53.300046 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aeaa687-c5ab-4385-a489-65b033853226 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:53.304900 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98459a4e-b48b-47af-8687-9f67b40df542 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:53.306108 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98459a4e-b48b-47af-8687-9f67b40df542 admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:53.307341 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98459a4e-b48b-47af-8687-9f67b40df542 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:53.307341 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98459a4e-b48b-47af-8687-9f67b40df542 admin admin] PATH_INFO: `/v3/projects/8263c43ce6824c8b8b9244ee66ff5ae3/users/ead580ae5e18434f9def54e46ea72149/roles/02e242d5520740d4b77a5f4008c7eb57` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:53.346506 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1020/2042] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:34:53 2022] PUT /identity/v3/projects/cac54fe0108d448abb161656c9c50bdd/users/e9e14e1e61c64e3aac538de88162d183/roles/02e242d5520740d4b77a5f4008c7eb57 => generated 0 bytes in 115 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:34:53.349385 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c66b164-79cf-47fa-8318-7f6400ecd38c None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:53.349385 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c66b164-79cf-47fa-8318-7f6400ecd38c None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:53.349760 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c66b164-79cf-47fa-8318-7f6400ecd38c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:53.368997 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98459a4e-b48b-47af-8687-9f67b40df542 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8263c43ce6824c8b8b9244ee66ff5ae3, user_id=ead580ae5e18434f9def54e46ea72149, role_id=02e242d5520740d4b77a5f4008c7eb57)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:53.380559 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98459a4e-b48b-47af-8687-9f67b40df542 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:53.421510 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-0c66b164-79cf-47fa-8318-7f6400ecd38c None None] MFA Rules not processed for user `e9e14e1e61c64e3aac538de88162d183`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:34:53.424258 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-0c66b164-79cf-47fa-8318-7f6400ecd38c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:53.437991 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1023/2043] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:34:53 2022] PUT /identity/v3/projects/8263c43ce6824c8b8b9244ee66ff5ae3/users/ead580ae5e18434f9def54e46ea72149/roles/02e242d5520740d4b77a5f4008c7eb57 => generated 0 bytes in 144 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:34:53.437991 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98459a4e-b48b-47af-8687-9f67b40df542 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:53.482904 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:34:53.537848 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:34:53.590887 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1021/2044] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:34:53 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 238 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:34:53.591336 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aefa8206-eacb-44c4-b8fa-23658091ceba None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:53.591877 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aefa8206-eacb-44c4-b8fa-23658091ceba None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:53.592364 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aefa8206-eacb-44c4-b8fa-23658091ceba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:53.642690 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98459a4e-b48b-47af-8687-9f67b40df542 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:53.650471 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdede29e-8cb8-4520-a81a-7e672b51cfb5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dc226f7ddcb49ecacb51ce57fdfc11b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:53.654370 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdede29e-8cb8-4520-a81a-7e672b51cfb5 service cinder] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:53.655415 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdede29e-8cb8-4520-a81a-7e672b51cfb5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:53.656100 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdede29e-8cb8-4520-a81a-7e672b51cfb5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:53.659694 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-fdede29e-8cb8-4520-a81a-7e672b51cfb5 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:53.661775 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-aefa8206-eacb-44c4-b8fa-23658091ceba None None] MFA Rules not processed for user `ead580ae5e18434f9def54e46ea72149`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:34:53.663316 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-aefa8206-eacb-44c4-b8fa-23658091ceba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:53.809906 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdede29e-8cb8-4520-a81a-7e672b51cfb5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:53.813020 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1022/2045] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:34:53 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 227 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:34:53.815063 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdede29e-8cb8-4520-a81a-7e672b51cfb5 service cinder] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:53.822056 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aefa8206-eacb-44c4-b8fa-23658091ceba None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:53.825244 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-aefa8206-eacb-44c4-b8fa-23658091ceba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:53.849609 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1024/2046] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:34:53 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 413 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:34:53.852573 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdede29e-8cb8-4520-a81a-7e672b51cfb5 service cinder] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:53.857208 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aefa8206-eacb-44c4-b8fa-23658091ceba None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:53.868848 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2305be0a-e50a-4268-984b-c0684422944f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dc226f7ddcb49ecacb51ce57fdfc11b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:53.870227 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2305be0a-e50a-4268-984b-c0684422944f service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:53.870750 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2305be0a-e50a-4268-984b-c0684422944f service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:53.871174 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2305be0a-e50a-4268-984b-c0684422944f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:53.891150 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2305be0a-e50a-4268-984b-c0684422944f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:53.894450 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2305be0a-e50a-4268-984b-c0684422944f service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:53.929366 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1023/2047] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:34:53 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 115 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:34:53.954776 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdede29e-8cb8-4520-a81a-7e672b51cfb5 service cinder] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:53.964521 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f64cc5c-a47a-4ec7-8fc0-45ec87d85c6b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:53.965312 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f64cc5c-a47a-4ec7-8fc0-45ec87d85c6b admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:53.965697 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f64cc5c-a47a-4ec7-8fc0-45ec87d85c6b admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:53.966103 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f64cc5c-a47a-4ec7-8fc0-45ec87d85c6b admin admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:53.966873 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f64cc5c-a47a-4ec7-8fc0-45ec87d85c6b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:53.968969 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f64cc5c-a47a-4ec7-8fc0-45ec87d85c6b admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:54.020964 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1025/2048] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:34:53 2022] POST /identity/v3/projects => generated 383 bytes in 171 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:34:54.025355 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2305be0a-e50a-4268-984b-c0684422944f service cinder] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:54.041491 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2305be0a-e50a-4268-984b-c0684422944f service cinder] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:54.221451 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fffc6c0b-1c0d-49ed-9a50-c411889e7ed5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:54.222636 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fffc6c0b-1c0d-49ed-9a50-c411889e7ed5 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:54.223831 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fffc6c0b-1c0d-49ed-9a50-c411889e7ed5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:54.226157 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fffc6c0b-1c0d-49ed-9a50-c411889e7ed5 admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:54.227204 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fffc6c0b-1c0d-49ed-9a50-c411889e7ed5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:54.231208 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fffc6c0b-1c0d-49ed-9a50-c411889e7ed5 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:54.276692 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1024/2049] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:34:54 2022] POST /identity/v3/users => generated 298 bytes in 252 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:34:54.283259 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f64cc5c-a47a-4ec7-8fc0-45ec87d85c6b admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:54.302292 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f64cc5c-a47a-4ec7-8fc0-45ec87d85c6b admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:54.309362 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3e2e054-a4b9-4b45-b2c6-1ccba2fd6c76 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:54.310248 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3e2e054-a4b9-4b45-b2c6-1ccba2fd6c76 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:54.311099 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3e2e054-a4b9-4b45-b2c6-1ccba2fd6c76 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:54.311509 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3e2e054-a4b9-4b45-b2c6-1ccba2fd6c76 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:54.312415 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3e2e054-a4b9-4b45-b2c6-1ccba2fd6c76 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:54.314250 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3e2e054-a4b9-4b45-b2c6-1ccba2fd6c76 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:54.335101 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1026/2050] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:34:54 2022] GET /identity/v3/roles => generated 1420 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:34:54.339987 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fffc6c0b-1c0d-49ed-9a50-c411889e7ed5 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:54.356045 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fffc6c0b-1c0d-49ed-9a50-c411889e7ed5 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:54.369440 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de95105d-ab29-4fab-9ed5-03a0194e2294 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:54.371090 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de95105d-ab29-4fab-9ed5-03a0194e2294 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:54.371837 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de95105d-ab29-4fab-9ed5-03a0194e2294 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:54.372529 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de95105d-ab29-4fab-9ed5-03a0194e2294 admin admin] PATH_INFO: `/v3/projects/92d71ebbf1d84444bb39977c4b6b9d09/users/5fa6fd38803a4aff89f19939cdb74b27/roles/a6e562cdc96c4be691d638990f8403fd` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:54.414339 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de95105d-ab29-4fab-9ed5-03a0194e2294 admin admin] RBAC: Authorizing `identity:create_grant(project_id=92d71ebbf1d84444bb39977c4b6b9d09, user_id=5fa6fd38803a4aff89f19939cdb74b27, role_id=a6e562cdc96c4be691d638990f8403fd)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:54.418157 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de95105d-ab29-4fab-9ed5-03a0194e2294 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:54.445932 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3e2e054-a4b9-4b45-b2c6-1ccba2fd6c76 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:54.447359 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1025/2051] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:34:54 2022] PUT /identity/v3/projects/92d71ebbf1d84444bb39977c4b6b9d09/users/5fa6fd38803a4aff89f19939cdb74b27/roles/a6e562cdc96c4be691d638990f8403fd => generated 0 bytes in 101 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:34:54.462983 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3e2e054-a4b9-4b45-b2c6-1ccba2fd6c76 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:54.649897 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:34:54.699831 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cad5389e-1f56-4811-9e67-3e1affd378b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:54.700903 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cad5389e-1f56-4811-9e67-3e1affd378b0 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:54.701401 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cad5389e-1f56-4811-9e67-3e1affd378b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:54.702005 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cad5389e-1f56-4811-9e67-3e1affd378b0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:54.703211 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cad5389e-1f56-4811-9e67-3e1affd378b0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:54.705273 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cad5389e-1f56-4811-9e67-3e1affd378b0 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:54.740318 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de95105d-ab29-4fab-9ed5-03a0194e2294 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:54.743922 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1027/2052] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:34:54 2022] GET /identity/v3/roles => generated 1420 bytes in 288 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:34:54.777075 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de95105d-ab29-4fab-9ed5-03a0194e2294 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:54.785550 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49288ca3-e5e0-4652-a015-4a04a9f1b225 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:54.786735 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49288ca3-e5e0-4652-a015-4a04a9f1b225 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:54.787124 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49288ca3-e5e0-4652-a015-4a04a9f1b225 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:54.787501 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49288ca3-e5e0-4652-a015-4a04a9f1b225 admin admin] PATH_INFO: `/v3/projects/92d71ebbf1d84444bb39977c4b6b9d09/users/5fa6fd38803a4aff89f19939cdb74b27/roles/a6e562cdc96c4be691d638990f8403fd` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:54.790547 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49288ca3-e5e0-4652-a015-4a04a9f1b225 admin admin] RBAC: Authorizing `identity:create_grant(project_id=92d71ebbf1d84444bb39977c4b6b9d09, user_id=5fa6fd38803a4aff89f19939cdb74b27, role_id=a6e562cdc96c4be691d638990f8403fd)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:54.794320 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49288ca3-e5e0-4652-a015-4a04a9f1b225 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:54.811224 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cad5389e-1f56-4811-9e67-3e1affd378b0 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:54.813041 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1026/2053] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:34:54 2022] PUT /identity/v3/projects/92d71ebbf1d84444bb39977c4b6b9d09/users/5fa6fd38803a4aff89f19939cdb74b27/roles/a6e562cdc96c4be691d638990f8403fd => generated 0 bytes in 74 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:34:54.826538 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-cad5389e-1f56-4811-9e67-3e1affd378b0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:54.837701 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6662f829-4ae3-4ef1-96b7-35f793134fc7 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:54.839471 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6662f829-4ae3-4ef1-96b7-35f793134fc7 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:54.844840 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cad5389e-1f56-4811-9e67-3e1affd378b0 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:54.845043 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6662f829-4ae3-4ef1-96b7-35f793134fc7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:54.908806 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:34:54.943815 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-6662f829-4ae3-4ef1-96b7-35f793134fc7 None None] MFA Rules not processed for user `5fa6fd38803a4aff89f19939cdb74b27`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:34:54.945253 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-6662f829-4ae3-4ef1-96b7-35f793134fc7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:54.950646 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50c4e53d-a618-44c5-9621-739a03ba05ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:54.951913 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c4e53d-a618-44c5-9621-739a03ba05ab admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:54.952091 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c4e53d-a618-44c5-9621-739a03ba05ab admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:54.952362 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c4e53d-a618-44c5-9621-739a03ba05ab admin admin] PATH_INFO: `/v3/projects/b0e9b513af904caf84105db15cd77dda` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:54.955491 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50c4e53d-a618-44c5-9621-739a03ba05ab admin admin] RBAC: Authorizing `identity:delete_project(project_id=b0e9b513af904caf84105db15cd77dda)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:54.958624 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50c4e53d-a618-44c5-9621-739a03ba05ab admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:55.042578 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-50c4e53d-a618-44c5-9621-739a03ba05ab admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b0e9b513af904caf84105db15cd77dda', 'request_id': 'req-50c4e53d-a618-44c5-9621-739a03ba05ab'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:55.055144 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-50c4e53d-a618-44c5-9621-739a03ba05ab admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b0e9b513af904caf84105db15cd77dda', 'request_id': 'req-50c4e53d-a618-44c5-9621-739a03ba05ab'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:55.067746 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-50c4e53d-a618-44c5-9621-739a03ba05ab admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b0e9b513af904caf84105db15cd77dda', 'request_id': 'req-50c4e53d-a618-44c5-9621-739a03ba05ab'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:55.083902 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1027/2054] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:34:54 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 258 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:34:55.095963 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-50c4e53d-a618-44c5-9621-739a03ba05ab admin admin] The token cache is being invalidate because project b0e9b513af904caf84105db15cd77dda was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81434) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:34:55.096793 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-50c4e53d-a618-44c5-9621-739a03ba05ab admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:55.097682 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-50c4e53d-a618-44c5-9621-739a03ba05ab admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:55.101215 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1028/2055] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:34:54 2022] DELETE /identity/v3/projects/b0e9b513af904caf84105db15cd77dda => generated 0 bytes in 291 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:34:55.111571 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50c4e53d-a618-44c5-9621-739a03ba05ab admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:55.115388 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-50c4e53d-a618-44c5-9621-739a03ba05ab admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:55.116849 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6662f829-4ae3-4ef1-96b7-35f793134fc7 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:55.134990 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6662f829-4ae3-4ef1-96b7-35f793134fc7 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:55.241460 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50c4e53d-a618-44c5-9621-739a03ba05ab admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:55.248177 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3675b0dc-82f5-48a6-936f-2b11faa1afb2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dc226f7ddcb49ecacb51ce57fdfc11b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:55.249306 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3675b0dc-82f5-48a6-936f-2b11faa1afb2 service cinder] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:55.249852 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3675b0dc-82f5-48a6-936f-2b11faa1afb2 service cinder] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:55.250394 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3675b0dc-82f5-48a6-936f-2b11faa1afb2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:55.253429 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-3675b0dc-82f5-48a6-936f-2b11faa1afb2 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:55.264305 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daec1e4d-d06f-4349-bc2b-5ad7bc761e38 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:55.265761 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daec1e4d-d06f-4349-bc2b-5ad7bc761e38 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:55.266254 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daec1e4d-d06f-4349-bc2b-5ad7bc761e38 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:55.266800 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daec1e4d-d06f-4349-bc2b-5ad7bc761e38 admin admin] PATH_INFO: `/v3/users/572ed224ebbc4e558a4c7c2ce8201bd5` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:55.268881 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daec1e4d-d06f-4349-bc2b-5ad7bc761e38 admin admin] RBAC: Authorizing `identity:delete_user(user_id=572ed224ebbc4e558a4c7c2ce8201bd5)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:55.272856 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daec1e4d-d06f-4349-bc2b-5ad7bc761e38 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:55.300791 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:34:55.404369 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3675b0dc-82f5-48a6-936f-2b11faa1afb2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:55.407977 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3675b0dc-82f5-48a6-936f-2b11faa1afb2 service cinder] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:55.416859 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-daec1e4d-d06f-4349-bc2b-5ad7bc761e38 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '572ed224ebbc4e558a4c7c2ce8201bd5'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:55.427969 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1029/2056] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:34:55 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 318 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:34:55.446055 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-daec1e4d-d06f-4349-bc2b-5ad7bc761e38 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '572ed224ebbc4e558a4c7c2ce8201bd5'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:55.446832 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-daec1e4d-d06f-4349-bc2b-5ad7bc761e38 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '572ed224ebbc4e558a4c7c2ce8201bd5'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:55.447499 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-daec1e4d-d06f-4349-bc2b-5ad7bc761e38 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '572ed224ebbc4e558a4c7c2ce8201bd5'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:55.456191 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-daec1e4d-d06f-4349-bc2b-5ad7bc761e38 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '572ed224ebbc4e558a4c7c2ce8201bd5'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:55.476688 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1028/2057] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:34:55 2022] DELETE /identity/v3/users/572ed224ebbc4e558a4c7c2ce8201bd5 => generated 0 bytes in 369 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:34:56.219467 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3675b0dc-82f5-48a6-936f-2b11faa1afb2 service cinder] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:56.221066 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-3675b0dc-82f5-48a6-936f-2b11faa1afb2 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:56.241272 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3675b0dc-82f5-48a6-936f-2b11faa1afb2 service cinder] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:56.366688 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03d233c1-70a5-422e-a475-eed9c2f1f3a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:56.368154 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03d233c1-70a5-422e-a475-eed9c2f1f3a0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:56.368763 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03d233c1-70a5-422e-a475-eed9c2f1f3a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:56.369999 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03d233c1-70a5-422e-a475-eed9c2f1f3a0 admin admin] PATH_INFO: `/v3/projects/cb3bc5833ea94b3bacca8826b813b662` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:56.372165 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03d233c1-70a5-422e-a475-eed9c2f1f3a0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=cb3bc5833ea94b3bacca8826b813b662)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:56.377324 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03d233c1-70a5-422e-a475-eed9c2f1f3a0 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:56.476353 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-03d233c1-70a5-422e-a475-eed9c2f1f3a0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cb3bc5833ea94b3bacca8826b813b662', 'request_id': 'req-03d233c1-70a5-422e-a475-eed9c2f1f3a0'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:56.485905 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-03d233c1-70a5-422e-a475-eed9c2f1f3a0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cb3bc5833ea94b3bacca8826b813b662', 'request_id': 'req-03d233c1-70a5-422e-a475-eed9c2f1f3a0'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:56.497764 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-03d233c1-70a5-422e-a475-eed9c2f1f3a0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cb3bc5833ea94b3bacca8826b813b662', 'request_id': 'req-03d233c1-70a5-422e-a475-eed9c2f1f3a0'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:56.516959 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-03d233c1-70a5-422e-a475-eed9c2f1f3a0 admin admin] The token cache is being invalidate because project cb3bc5833ea94b3bacca8826b813b662 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81434) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:34:56.517462 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-03d233c1-70a5-422e-a475-eed9c2f1f3a0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:56.518811 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-03d233c1-70a5-422e-a475-eed9c2f1f3a0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:34:56.521615 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1030/2058] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:34:56 2022] DELETE /identity/v3/projects/cb3bc5833ea94b3bacca8826b813b662 => generated 0 bytes in 304 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 21:34:56.528859 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6af533b1-f7f0-4698-ac5d-7d1a3c625b09 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:56.529337 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6af533b1-f7f0-4698-ac5d-7d1a3c625b09 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:56.529833 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6af533b1-f7f0-4698-ac5d-7d1a3c625b09 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:56.563749 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-6af533b1-f7f0-4698-ac5d-7d1a3c625b09 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:34:56.565774 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-6af533b1-f7f0-4698-ac5d-7d1a3c625b09 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:56.582922 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:34:56.679177 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1029/2059] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:34:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 152 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:34:56.687521 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2b3a947-47aa-477c-8cb2-35f95d38aa18 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:56.687795 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2b3a947-47aa-477c-8cb2-35f95d38aa18 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:56.688068 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2b3a947-47aa-477c-8cb2-35f95d38aa18 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:56.716818 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-a2b3a947-47aa-477c-8cb2-35f95d38aa18 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:34:56.718192 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-a2b3a947-47aa-477c-8cb2-35f95d38aa18 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:56.729442 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1031/2060] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:34:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 43 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 21:34:56.736078 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6af533b1-f7f0-4698-ac5d-7d1a3c625b09 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:56.748718 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6af533b1-f7f0-4698-ac5d-7d1a3c625b09 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:56.753606 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afea0c30-47fe-4b86-a82b-e5e6be8bc182 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:56.754851 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afea0c30-47fe-4b86-a82b-e5e6be8bc182 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:56.755137 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afea0c30-47fe-4b86-a82b-e5e6be8bc182 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:56.755415 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afea0c30-47fe-4b86-a82b-e5e6be8bc182 admin admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:56.756083 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afea0c30-47fe-4b86-a82b-e5e6be8bc182 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:34:56.756398 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afea0c30-47fe-4b86-a82b-e5e6be8bc182 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:56.758209 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afea0c30-47fe-4b86-a82b-e5e6be8bc182 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:56.773135 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1030/2061] 10.222.0.90 () {58 vars in 1153 bytes} [Mon Aug 29 21:34:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:34:56.777479 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2b3a947-47aa-477c-8cb2-35f95d38aa18 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:56.790402 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2b3a947-47aa-477c-8cb2-35f95d38aa18 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:56.797156 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c718aea-d83e-4ed7-a4b1-b73733bea5dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:56.798052 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c718aea-d83e-4ed7-a4b1-b73733bea5dc admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:56.798388 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c718aea-d83e-4ed7-a4b1-b73733bea5dc admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:56.798710 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c718aea-d83e-4ed7-a4b1-b73733bea5dc admin admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:56.799831 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c718aea-d83e-4ed7-a4b1-b73733bea5dc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:56.803018 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c718aea-d83e-4ed7-a4b1-b73733bea5dc admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:56.850425 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1032/2062] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:34:56 2022] POST /identity/v3/projects => generated 385 bytes in 73 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:34:56.858111 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afea0c30-47fe-4b86-a82b-e5e6be8bc182 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:56.869412 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afea0c30-47fe-4b86-a82b-e5e6be8bc182 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:56.937185 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-722079f9-271a-481a-8ef5-23ba48c1b12a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:56.937889 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-722079f9-271a-481a-8ef5-23ba48c1b12a admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:56.938210 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-722079f9-271a-481a-8ef5-23ba48c1b12a admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:56.938546 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-722079f9-271a-481a-8ef5-23ba48c1b12a admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:56.939263 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-722079f9-271a-481a-8ef5-23ba48c1b12a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:56.941256 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-722079f9-271a-481a-8ef5-23ba48c1b12a admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:56.967638 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1031/2063] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:34:56 2022] POST /identity/v3/users => generated 293 bytes in 112 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 21:34:56.974399 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c718aea-d83e-4ed7-a4b1-b73733bea5dc admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:56.993757 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c718aea-d83e-4ed7-a4b1-b73733bea5dc admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:57.000040 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7a354a1-798e-434a-8796-fd699f41fff1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:57.000757 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7a354a1-798e-434a-8796-fd699f41fff1 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:57.001100 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7a354a1-798e-434a-8796-fd699f41fff1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:57.001545 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7a354a1-798e-434a-8796-fd699f41fff1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:57.002382 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7a354a1-798e-434a-8796-fd699f41fff1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:57.005011 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7a354a1-798e-434a-8796-fd699f41fff1 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:57.016199 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1033/2064] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:34:56 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:34:57.022545 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-722079f9-271a-481a-8ef5-23ba48c1b12a admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:57.034990 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-722079f9-271a-481a-8ef5-23ba48c1b12a admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:57.040470 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ac621f4-35f3-48d5-a653-c2cef6bf35e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:57.041176 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ac621f4-35f3-48d5-a653-c2cef6bf35e6 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:57.041457 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ac621f4-35f3-48d5-a653-c2cef6bf35e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:57.041731 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ac621f4-35f3-48d5-a653-c2cef6bf35e6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:57.042424 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ac621f4-35f3-48d5-a653-c2cef6bf35e6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:57.044042 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ac621f4-35f3-48d5-a653-c2cef6bf35e6 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:57.056249 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1032/2065] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:34:57 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:34:57.062491 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7a354a1-798e-434a-8796-fd699f41fff1 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:57.081097 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7a354a1-798e-434a-8796-fd699f41fff1 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:57.089193 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a06c8508-a364-454e-9b6c-e9bb9523c780 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:57.090331 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a06c8508-a364-454e-9b6c-e9bb9523c780 admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:57.090793 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a06c8508-a364-454e-9b6c-e9bb9523c780 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:57.091244 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a06c8508-a364-454e-9b6c-e9bb9523c780 admin admin] PATH_INFO: `/v3/projects/c54de3a6284c4742b8c30e0021f71f00/users/2444cf2966d94c3f9cf61774ce88e48d/roles/02e242d5520740d4b77a5f4008c7eb57` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:57.111169 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:34:57.126478 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a06c8508-a364-454e-9b6c-e9bb9523c780 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c54de3a6284c4742b8c30e0021f71f00, user_id=2444cf2966d94c3f9cf61774ce88e48d, role_id=02e242d5520740d4b77a5f4008c7eb57)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:57.133945 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a06c8508-a364-454e-9b6c-e9bb9523c780 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:57.159493 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1034/2066] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:34:57 2022] PUT /identity/v3/projects/c54de3a6284c4742b8c30e0021f71f00/users/2444cf2966d94c3f9cf61774ce88e48d/roles/02e242d5520740d4b77a5f4008c7eb57 => generated 0 bytes in 98 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:34:57.173003 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c1ec4c5-5db8-4e52-ab29-cce5a1c207c0 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:57.173495 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c1ec4c5-5db8-4e52-ab29-cce5a1c207c0 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:57.173983 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c1ec4c5-5db8-4e52-ab29-cce5a1c207c0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:57.247405 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-1c1ec4c5-5db8-4e52-ab29-cce5a1c207c0 None None] MFA Rules not processed for user `2444cf2966d94c3f9cf61774ce88e48d`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:34:57.249538 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-1c1ec4c5-5db8-4e52-ab29-cce5a1c207c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:57.370087 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1033/2067] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:34:57 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 199 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:34:57.394438 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a06c8508-a364-454e-9b6c-e9bb9523c780 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:57.397157 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-a06c8508-a364-454e-9b6c-e9bb9523c780 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:57.479069 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a06c8508-a364-454e-9b6c-e9bb9523c780 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:57.483733 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-520a28e8-e888-4acc-b207-3b4004e3f18c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dc226f7ddcb49ecacb51ce57fdfc11b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:57.484745 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-520a28e8-e888-4acc-b207-3b4004e3f18c service cinder] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:57.493630 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-520a28e8-e888-4acc-b207-3b4004e3f18c service cinder] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:57.494175 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-520a28e8-e888-4acc-b207-3b4004e3f18c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:57.510551 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-520a28e8-e888-4acc-b207-3b4004e3f18c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:57.512013 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-520a28e8-e888-4acc-b207-3b4004e3f18c service cinder] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:57.529692 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1035/2068] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:34:57 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 136 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:34:57.773931 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c1ec4c5-5db8-4e52-ab29-cce5a1c207c0 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:57.776135 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-1c1ec4c5-5db8-4e52-ab29-cce5a1c207c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:57.796535 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c1ec4c5-5db8-4e52-ab29-cce5a1c207c0 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:57.802813 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9bd90f5-4902-42ec-b378-41d91ae83fac service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dc226f7ddcb49ecacb51ce57fdfc11b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:57.803681 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9bd90f5-4902-42ec-b378-41d91ae83fac service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:57.804184 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9bd90f5-4902-42ec-b378-41d91ae83fac service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:57.804657 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9bd90f5-4902-42ec-b378-41d91ae83fac service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:57.807399 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-b9bd90f5-4902-42ec-b378-41d91ae83fac service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:34:57.821374 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:34:57.930934 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9bd90f5-4902-42ec-b378-41d91ae83fac service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:34:57.935892 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9bd90f5-4902-42ec-b378-41d91ae83fac service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:34:57.961995 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1034/2069] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:34:57 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 189 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:34:59.890625 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-520a28e8-e888-4acc-b207-3b4004e3f18c service cinder] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:34:59.903096 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-520a28e8-e888-4acc-b207-3b4004e3f18c service cinder] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:34:59.908867 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6dac83c-5de6-4d66-984c-55a0b7fcd5ef service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dc226f7ddcb49ecacb51ce57fdfc11b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:34:59.908867 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6dac83c-5de6-4d66-984c-55a0b7fcd5ef service cinder] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:34:59.908867 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6dac83c-5de6-4d66-984c-55a0b7fcd5ef service cinder] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:34:59.909194 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6dac83c-5de6-4d66-984c-55a0b7fcd5ef service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:34:59.915000 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-f6dac83c-5de6-4d66-984c-55a0b7fcd5ef service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:00.016014 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6dac83c-5de6-4d66-984c-55a0b7fcd5ef service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:00.017684 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6dac83c-5de6-4d66-984c-55a0b7fcd5ef service cinder] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:00.033073 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1036/2070] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:34:59 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 143 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:35:00.559007 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9bd90f5-4902-42ec-b378-41d91ae83fac service cinder] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:00.560779 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-b9bd90f5-4902-42ec-b378-41d91ae83fac service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:00.642087 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9bd90f5-4902-42ec-b378-41d91ae83fac service cinder] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:00.648433 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f31c061-4f6f-40d3-aee2-9e52340c5dde admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:00.649077 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f31c061-4f6f-40d3-aee2-9e52340c5dde admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:00.649355 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f31c061-4f6f-40d3-aee2-9e52340c5dde admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:00.649625 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f31c061-4f6f-40d3-aee2-9e52340c5dde admin admin] PATH_INFO: `/v3/users/0782c69725e54da597c140a55ffd6e89` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:00.650782 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f31c061-4f6f-40d3-aee2-9e52340c5dde admin admin] RBAC: Authorizing `identity:delete_user(user_id=0782c69725e54da597c140a55ffd6e89)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:00.652706 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f31c061-4f6f-40d3-aee2-9e52340c5dde admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:00.727905 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-6f31c061-4f6f-40d3-aee2-9e52340c5dde admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0782c69725e54da597c140a55ffd6e89'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:00.747776 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-6f31c061-4f6f-40d3-aee2-9e52340c5dde admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0782c69725e54da597c140a55ffd6e89'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:00.748557 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-6f31c061-4f6f-40d3-aee2-9e52340c5dde admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0782c69725e54da597c140a55ffd6e89'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:00.749204 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-6f31c061-4f6f-40d3-aee2-9e52340c5dde admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0782c69725e54da597c140a55ffd6e89'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:00.758321 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-6f31c061-4f6f-40d3-aee2-9e52340c5dde admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0782c69725e54da597c140a55ffd6e89'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:00.788676 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1035/2071] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:35:00 2022] DELETE /identity/v3/users/0782c69725e54da597c140a55ffd6e89 => generated 0 bytes in 230 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 21:35:00.793690 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f3d0885-0ce1-491a-835d-1eea2f3fc8e4 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:00.793936 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f3d0885-0ce1-491a-835d-1eea2f3fc8e4 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:00.794201 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f3d0885-0ce1-491a-835d-1eea2f3fc8e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:00.814369 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-4f3d0885-0ce1-491a-835d-1eea2f3fc8e4 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:35:00.815620 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-4f3d0885-0ce1-491a-835d-1eea2f3fc8e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:00.906077 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1037/2072] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:35:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:35:00.921724 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f31c061-4f6f-40d3-aee2-9e52340c5dde admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:00.923292 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-6f31c061-4f6f-40d3-aee2-9e52340c5dde admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:00.985124 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:35:01.033670 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f31c061-4f6f-40d3-aee2-9e52340c5dde admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:01.039666 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0dda9ea-a7e8-4ccd-840a-10140368126e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:01.040585 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0dda9ea-a7e8-4ccd-840a-10140368126e service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:01.041060 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0dda9ea-a7e8-4ccd-840a-10140368126e service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:01.046286 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0dda9ea-a7e8-4ccd-840a-10140368126e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:01.069391 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0dda9ea-a7e8-4ccd-840a-10140368126e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:01.072198 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0dda9ea-a7e8-4ccd-840a-10140368126e service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:01.102387 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1036/2073] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:35:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 181 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:35:01.543982 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f3d0885-0ce1-491a-835d-1eea2f3fc8e4 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:01.546879 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-4f3d0885-0ce1-491a-835d-1eea2f3fc8e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:01.577353 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f3d0885-0ce1-491a-835d-1eea2f3fc8e4 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:01.607601 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1851cca7-040f-4124-8933-c5f09e121aa5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:01.609151 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1851cca7-040f-4124-8933-c5f09e121aa5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:01.612788 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1851cca7-040f-4124-8933-c5f09e121aa5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:01.614855 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1851cca7-040f-4124-8933-c5f09e121aa5 admin admin] PATH_INFO: `/v3/projects/9b54c513066a4e8289eec3a708d82256` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:01.627975 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1851cca7-040f-4124-8933-c5f09e121aa5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9b54c513066a4e8289eec3a708d82256)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:01.634219 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1851cca7-040f-4124-8933-c5f09e121aa5 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:01.681860 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:35:01.745932 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-1851cca7-040f-4124-8933-c5f09e121aa5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9b54c513066a4e8289eec3a708d82256', 'request_id': 'req-1851cca7-040f-4124-8933-c5f09e121aa5'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:01.756344 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-1851cca7-040f-4124-8933-c5f09e121aa5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9b54c513066a4e8289eec3a708d82256', 'request_id': 'req-1851cca7-040f-4124-8933-c5f09e121aa5'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:01.765509 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-1851cca7-040f-4124-8933-c5f09e121aa5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9b54c513066a4e8289eec3a708d82256', 'request_id': 'req-1851cca7-040f-4124-8933-c5f09e121aa5'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:01.783329 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-1851cca7-040f-4124-8933-c5f09e121aa5 admin admin] The token cache is being invalidate because project 9b54c513066a4e8289eec3a708d82256 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81434) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:35:01.783629 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-1851cca7-040f-4124-8933-c5f09e121aa5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:01.784470 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-1851cca7-040f-4124-8933-c5f09e121aa5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:01.787704 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1038/2074] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:35:01 2022] DELETE /identity/v3/projects/9b54c513066a4e8289eec3a708d82256 => generated 0 bytes in 245 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 21:35:01.793812 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0dda9ea-a7e8-4ccd-840a-10140368126e service neutron] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:01.812427 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0dda9ea-a7e8-4ccd-840a-10140368126e service neutron] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:01.903058 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08e53f5c-f8ac-40a7-aabb-bf7aa6267530 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:01.903999 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08e53f5c-f8ac-40a7-aabb-bf7aa6267530 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:01.904482 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08e53f5c-f8ac-40a7-aabb-bf7aa6267530 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:01.904833 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08e53f5c-f8ac-40a7-aabb-bf7aa6267530 admin admin] PATH_INFO: `/v3/users/22c1b225b1044334a2de2761d584f048` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:01.906362 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08e53f5c-f8ac-40a7-aabb-bf7aa6267530 admin admin] RBAC: Authorizing `identity:delete_user(user_id=22c1b225b1044334a2de2761d584f048)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:01.910284 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08e53f5c-f8ac-40a7-aabb-bf7aa6267530 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:01.985398 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-08e53f5c-f8ac-40a7-aabb-bf7aa6267530 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '22c1b225b1044334a2de2761d584f048'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:02.003458 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-08e53f5c-f8ac-40a7-aabb-bf7aa6267530 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '22c1b225b1044334a2de2761d584f048'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:02.005629 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-08e53f5c-f8ac-40a7-aabb-bf7aa6267530 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '22c1b225b1044334a2de2761d584f048'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:02.006369 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-08e53f5c-f8ac-40a7-aabb-bf7aa6267530 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '22c1b225b1044334a2de2761d584f048'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:02.018005 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-08e53f5c-f8ac-40a7-aabb-bf7aa6267530 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '22c1b225b1044334a2de2761d584f048'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:02.034489 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1037/2075] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:35:01 2022] DELETE /identity/v3/users/22c1b225b1044334a2de2761d584f048 => generated 0 bytes in 242 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 21:35:02.513877 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1851cca7-040f-4124-8933-c5f09e121aa5 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:02.516179 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-1851cca7-040f-4124-8933-c5f09e121aa5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:02.547610 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1851cca7-040f-4124-8933-c5f09e121aa5 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:02.647658 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5f8fa31-1fa5-4a33-954f-b99459cb1e85 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:02.648430 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5f8fa31-1fa5-4a33-954f-b99459cb1e85 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:02.648853 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5f8fa31-1fa5-4a33-954f-b99459cb1e85 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:02.649263 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5f8fa31-1fa5-4a33-954f-b99459cb1e85 admin admin] PATH_INFO: `/v3/projects/dea618c99b294c0ba9e8484dcd731e73` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:02.650850 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5f8fa31-1fa5-4a33-954f-b99459cb1e85 admin admin] RBAC: Authorizing `identity:delete_project(project_id=dea618c99b294c0ba9e8484dcd731e73)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:02.653191 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5f8fa31-1fa5-4a33-954f-b99459cb1e85 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:02.722139 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-d5f8fa31-1fa5-4a33-954f-b99459cb1e85 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'dea618c99b294c0ba9e8484dcd731e73', 'request_id': 'req-d5f8fa31-1fa5-4a33-954f-b99459cb1e85'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:02.742323 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-d5f8fa31-1fa5-4a33-954f-b99459cb1e85 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'dea618c99b294c0ba9e8484dcd731e73', 'request_id': 'req-d5f8fa31-1fa5-4a33-954f-b99459cb1e85'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:02.757366 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-d5f8fa31-1fa5-4a33-954f-b99459cb1e85 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'dea618c99b294c0ba9e8484dcd731e73', 'request_id': 'req-d5f8fa31-1fa5-4a33-954f-b99459cb1e85'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:02.778442 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-d5f8fa31-1fa5-4a33-954f-b99459cb1e85 admin admin] The token cache is being invalidate because project dea618c99b294c0ba9e8484dcd731e73 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81434) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:35:02.778797 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-d5f8fa31-1fa5-4a33-954f-b99459cb1e85 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:02.779566 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-d5f8fa31-1fa5-4a33-954f-b99459cb1e85 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:02.781864 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1039/2076] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:35:02 2022] DELETE /identity/v3/projects/dea618c99b294c0ba9e8484dcd731e73 => generated 0 bytes in 268 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 21:35:02.792280 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d9b9680-e4f6-4cc2-bb40-8104096b4b5f None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:02.792701 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d9b9680-e4f6-4cc2-bb40-8104096b4b5f None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:02.793237 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d9b9680-e4f6-4cc2-bb40-8104096b4b5f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:02.827547 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-7d9b9680-e4f6-4cc2-bb40-8104096b4b5f None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:35:02.829732 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-7d9b9680-e4f6-4cc2-bb40-8104096b4b5f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:02.898018 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:35:02.924479 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1038/2077] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:35:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:35:02.933640 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e7a1304-ee6d-4bd1-9496-93b087c8e5b2 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:02.933968 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e7a1304-ee6d-4bd1-9496-93b087c8e5b2 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:02.934255 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e7a1304-ee6d-4bd1-9496-93b087c8e5b2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:02.988710 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-8e7a1304-ee6d-4bd1-9496-93b087c8e5b2 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:35:02.990345 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-8e7a1304-ee6d-4bd1-9496-93b087c8e5b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:03.007280 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1040/2078] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:35:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 75 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:35:03.013818 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d9b9680-e4f6-4cc2-bb40-8104096b4b5f None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:03.031833 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d9b9680-e4f6-4cc2-bb40-8104096b4b5f None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:03.043735 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0d559c6-2cda-4f8e-bd7c-a8d8d79b8583 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:03.044776 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0d559c6-2cda-4f8e-bd7c-a8d8d79b8583 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:03.045241 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0d559c6-2cda-4f8e-bd7c-a8d8d79b8583 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:03.045724 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0d559c6-2cda-4f8e-bd7c-a8d8d79b8583 admin admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:03.046933 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0d559c6-2cda-4f8e-bd7c-a8d8d79b8583 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:35:03.047469 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0d559c6-2cda-4f8e-bd7c-a8d8d79b8583 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:03.050521 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0d559c6-2cda-4f8e-bd7c-a8d8d79b8583 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:03.065775 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1039/2079] 10.222.0.90 () {58 vars in 1153 bytes} [Mon Aug 29 21:35:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:35:03.071054 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e7a1304-ee6d-4bd1-9496-93b087c8e5b2 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:03.093223 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e7a1304-ee6d-4bd1-9496-93b087c8e5b2 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:03.101904 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f80c28bc-8c9e-4dc6-bad7-b95a422821cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:03.103199 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f80c28bc-8c9e-4dc6-bad7-b95a422821cc admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:03.103772 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f80c28bc-8c9e-4dc6-bad7-b95a422821cc admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:03.104280 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f80c28bc-8c9e-4dc6-bad7-b95a422821cc admin admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:03.105514 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f80c28bc-8c9e-4dc6-bad7-b95a422821cc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:03.109186 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f80c28bc-8c9e-4dc6-bad7-b95a422821cc admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:03.151692 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1041/2080] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:35:03 2022] POST /identity/v3/projects => generated 383 bytes in 81 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 21:35:03.156187 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0d559c6-2cda-4f8e-bd7c-a8d8d79b8583 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:03.175519 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0d559c6-2cda-4f8e-bd7c-a8d8d79b8583 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:03.243741 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d384bade-c188-43d6-86e1-c832a0c33fd0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:03.245034 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d384bade-c188-43d6-86e1-c832a0c33fd0 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:03.245464 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d384bade-c188-43d6-86e1-c832a0c33fd0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:03.245977 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d384bade-c188-43d6-86e1-c832a0c33fd0 admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:03.246693 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d384bade-c188-43d6-86e1-c832a0c33fd0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:03.248722 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d384bade-c188-43d6-86e1-c832a0c33fd0 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:03.282390 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1040/2081] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:35:03 2022] POST /identity/v3/users => generated 292 bytes in 123 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:35:03.283875 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f80c28bc-8c9e-4dc6-bad7-b95a422821cc admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:03.299655 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f80c28bc-8c9e-4dc6-bad7-b95a422821cc admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:03.305840 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b135827-885d-46a0-a9f1-66a2ffc26cf7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:03.306893 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b135827-885d-46a0-a9f1-66a2ffc26cf7 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:03.307424 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b135827-885d-46a0-a9f1-66a2ffc26cf7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:03.307907 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b135827-885d-46a0-a9f1-66a2ffc26cf7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:03.309238 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b135827-885d-46a0-a9f1-66a2ffc26cf7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:03.312216 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b135827-885d-46a0-a9f1-66a2ffc26cf7 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:03.323043 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:DELETE=14,keystone:SELECT=83,keystone:INSERT=3 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:35:03.333502 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1042/2082] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:35:03 2022] GET /identity/v3/roles => generated 1420 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 29 21:35:03.339296 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d384bade-c188-43d6-86e1-c832a0c33fd0 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:03.356414 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d384bade-c188-43d6-86e1-c832a0c33fd0 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:03.361747 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-734ae968-fd81-403a-8ed8-d1528a3074a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:03.362530 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-734ae968-fd81-403a-8ed8-d1528a3074a2 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:03.362923 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-734ae968-fd81-403a-8ed8-d1528a3074a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:03.363327 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-734ae968-fd81-403a-8ed8-d1528a3074a2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:03.364128 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-734ae968-fd81-403a-8ed8-d1528a3074a2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:03.366048 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-734ae968-fd81-403a-8ed8-d1528a3074a2 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:03.381575 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1041/2083] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:35:03 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 29 21:35:03.385277 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b135827-885d-46a0-a9f1-66a2ffc26cf7 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:03.398346 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b135827-885d-46a0-a9f1-66a2ffc26cf7 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:03.404386 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aca41ab3-f01f-4d7a-87e6-a4cf5442a81a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:03.404985 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aca41ab3-f01f-4d7a-87e6-a4cf5442a81a admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:03.405262 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aca41ab3-f01f-4d7a-87e6-a4cf5442a81a admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:03.405550 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aca41ab3-f01f-4d7a-87e6-a4cf5442a81a admin admin] PATH_INFO: `/v3/projects/5c215d2f45d8468a870b116e180233d0/users/1d27e52b1f1b4cf990b53ac23b15dc6f/roles/02e242d5520740d4b77a5f4008c7eb57` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:03.431586 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aca41ab3-f01f-4d7a-87e6-a4cf5442a81a admin admin] RBAC: Authorizing `identity:create_grant(project_id=5c215d2f45d8468a870b116e180233d0, user_id=1d27e52b1f1b4cf990b53ac23b15dc6f, role_id=02e242d5520740d4b77a5f4008c7eb57)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:03.434464 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aca41ab3-f01f-4d7a-87e6-a4cf5442a81a admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:03.456653 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1043/2084] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:35:03 2022] PUT /identity/v3/projects/5c215d2f45d8468a870b116e180233d0/users/1d27e52b1f1b4cf990b53ac23b15dc6f/roles/02e242d5520740d4b77a5f4008c7eb57 => generated 0 bytes in 72 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:35:03.469010 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29cf3d7d-bfae-48fa-a984-b0528b4d5f26 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:03.469370 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29cf3d7d-bfae-48fa-a984-b0528b4d5f26 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:03.469673 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29cf3d7d-bfae-48fa-a984-b0528b4d5f26 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:03.511163 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-29cf3d7d-bfae-48fa-a984-b0528b4d5f26 None None] MFA Rules not processed for user `1d27e52b1f1b4cf990b53ac23b15dc6f`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:35:03.512815 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-29cf3d7d-bfae-48fa-a984-b0528b4d5f26 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:03.618803 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1042/2085] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:35:03 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:35:03.620338 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aca41ab3-f01f-4d7a-87e6-a4cf5442a81a admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:03.732159 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aca41ab3-f01f-4d7a-87e6-a4cf5442a81a admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:03.739869 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccadac90-9fcd-403d-903d-fa9d05506ad3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:03.740864 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccadac90-9fcd-403d-903d-fa9d05506ad3 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:03.741329 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccadac90-9fcd-403d-903d-fa9d05506ad3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:03.741867 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccadac90-9fcd-403d-903d-fa9d05506ad3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:03.742976 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccadac90-9fcd-403d-903d-fa9d05506ad3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:03.746443 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccadac90-9fcd-403d-903d-fa9d05506ad3 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:03.812293 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1044/2086] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:35:03 2022] POST /identity/v3/projects => generated 385 bytes in 193 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:35:03.821227 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29cf3d7d-bfae-48fa-a984-b0528b4d5f26 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:03.869581 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29cf3d7d-bfae-48fa-a984-b0528b4d5f26 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:03.928848 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:35:03.952061 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-211cf6bd-90f8-4c6b-97d8-e82e3d466401 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:03.952708 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-211cf6bd-90f8-4c6b-97d8-e82e3d466401 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:03.952999 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-211cf6bd-90f8-4c6b-97d8-e82e3d466401 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:03.953266 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-211cf6bd-90f8-4c6b-97d8-e82e3d466401 admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:03.953954 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-211cf6bd-90f8-4c6b-97d8-e82e3d466401 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:03.956264 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-211cf6bd-90f8-4c6b-97d8-e82e3d466401 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:03.989071 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1043/2087] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:35:03 2022] POST /identity/v3/users => generated 293 bytes in 165 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:35:03.991889 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccadac90-9fcd-403d-903d-fa9d05506ad3 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:04.007598 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccadac90-9fcd-403d-903d-fa9d05506ad3 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:04.030497 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0db70b0-27d0-4dd9-8bde-9942610b7224 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:04.031250 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0db70b0-27d0-4dd9-8bde-9942610b7224 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:04.031745 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0db70b0-27d0-4dd9-8bde-9942610b7224 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:04.032282 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0db70b0-27d0-4dd9-8bde-9942610b7224 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:04.033323 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0db70b0-27d0-4dd9-8bde-9942610b7224 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:04.035715 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0db70b0-27d0-4dd9-8bde-9942610b7224 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:04.065517 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1045/2088] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:35:03 2022] GET /identity/v3/roles => generated 1420 bytes in 75 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:35:04.071553 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-211cf6bd-90f8-4c6b-97d8-e82e3d466401 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:04.088881 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-211cf6bd-90f8-4c6b-97d8-e82e3d466401 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:04.103056 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb21cea4-6a72-4776-9161-798cede5a3fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:04.104053 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb21cea4-6a72-4776-9161-798cede5a3fc admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:04.104421 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb21cea4-6a72-4776-9161-798cede5a3fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:04.104917 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb21cea4-6a72-4776-9161-798cede5a3fc admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:04.105812 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb21cea4-6a72-4776-9161-798cede5a3fc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:04.108074 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb21cea4-6a72-4776-9161-798cede5a3fc admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:04.120704 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1044/2089] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:35:04 2022] GET /identity/v3/roles => generated 1420 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 29 21:35:04.125697 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0db70b0-27d0-4dd9-8bde-9942610b7224 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:04.141302 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0db70b0-27d0-4dd9-8bde-9942610b7224 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:04.148541 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc123b7e-b334-4161-8ee7-9625ea8543a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:04.149397 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc123b7e-b334-4161-8ee7-9625ea8543a1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:04.149760 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc123b7e-b334-4161-8ee7-9625ea8543a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:04.150157 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc123b7e-b334-4161-8ee7-9625ea8543a1 admin admin] PATH_INFO: `/v3/projects/0034b8df1b904a0a92c173efa4b32fb8/users/8f466fa1017840d58019b0d2d2ce738c/roles/02e242d5520740d4b77a5f4008c7eb57` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:04.167310 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc123b7e-b334-4161-8ee7-9625ea8543a1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0034b8df1b904a0a92c173efa4b32fb8, user_id=8f466fa1017840d58019b0d2d2ce738c, role_id=02e242d5520740d4b77a5f4008c7eb57)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:04.170032 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc123b7e-b334-4161-8ee7-9625ea8543a1 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:04.186993 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1046/2090] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:35:04 2022] PUT /identity/v3/projects/0034b8df1b904a0a92c173efa4b32fb8/users/8f466fa1017840d58019b0d2d2ce738c/roles/02e242d5520740d4b77a5f4008c7eb57 => generated 0 bytes in 62 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:35:04.205367 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3bc137e-c2f7-4820-91fb-d27a55e601e9 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:04.205657 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3bc137e-c2f7-4820-91fb-d27a55e601e9 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:04.205967 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3bc137e-c2f7-4820-91fb-d27a55e601e9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:04.247598 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-b3bc137e-c2f7-4820-91fb-d27a55e601e9 None None] MFA Rules not processed for user `8f466fa1017840d58019b0d2d2ce738c`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:35:04.249102 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-b3bc137e-c2f7-4820-91fb-d27a55e601e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:04.356342 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1045/2091] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:35:04 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 152 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 21:35:04.361273 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc123b7e-b334-4161-8ee7-9625ea8543a1 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:04.491497 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc123b7e-b334-4161-8ee7-9625ea8543a1 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:04.498257 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47f7f1e2-eefb-4f46-a933-2d0486d086d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:04.498900 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f7f1e2-eefb-4f46-a933-2d0486d086d7 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:04.499179 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f7f1e2-eefb-4f46-a933-2d0486d086d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:04.499460 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f7f1e2-eefb-4f46-a933-2d0486d086d7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:04.500777 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47f7f1e2-eefb-4f46-a933-2d0486d086d7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:04.502753 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47f7f1e2-eefb-4f46-a933-2d0486d086d7 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:04.559327 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1047/2092] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:35:04 2022] POST /identity/v3/projects => generated 385 bytes in 199 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:35:04.565318 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3bc137e-c2f7-4820-91fb-d27a55e601e9 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:04.585830 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3bc137e-c2f7-4820-91fb-d27a55e601e9 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:04.692882 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-566ed53b-6607-45b4-8363-3c09ef895a84 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:04.693806 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-566ed53b-6607-45b4-8363-3c09ef895a84 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:04.694475 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-566ed53b-6607-45b4-8363-3c09ef895a84 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:04.695725 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-566ed53b-6607-45b4-8363-3c09ef895a84 admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:04.696841 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-566ed53b-6607-45b4-8363-3c09ef895a84 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:04.699085 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-566ed53b-6607-45b4-8363-3c09ef895a84 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:04.739325 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1046/2093] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:35:04 2022] POST /identity/v3/users => generated 299 bytes in 175 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:35:04.745277 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47f7f1e2-eefb-4f46-a933-2d0486d086d7 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:04.771024 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47f7f1e2-eefb-4f46-a933-2d0486d086d7 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:04.779289 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29c702b5-51a3-4e1d-aba4-e41236cdb335 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:04.780358 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29c702b5-51a3-4e1d-aba4-e41236cdb335 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:04.783373 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29c702b5-51a3-4e1d-aba4-e41236cdb335 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:04.783900 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29c702b5-51a3-4e1d-aba4-e41236cdb335 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:04.785194 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29c702b5-51a3-4e1d-aba4-e41236cdb335 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:04.788728 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29c702b5-51a3-4e1d-aba4-e41236cdb335 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:04.800024 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:35:04.814936 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1048/2094] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:35:04 2022] GET /identity/v3/roles => generated 1420 bytes in 71 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:35:04.818696 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-566ed53b-6607-45b4-8363-3c09ef895a84 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:04.839615 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-566ed53b-6607-45b4-8363-3c09ef895a84 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:04.846415 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1f96b15-fbe8-4c5f-8432-2ce952b87a30 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:04.848091 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1f96b15-fbe8-4c5f-8432-2ce952b87a30 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:04.848841 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1f96b15-fbe8-4c5f-8432-2ce952b87a30 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:04.849598 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1f96b15-fbe8-4c5f-8432-2ce952b87a30 admin admin] PATH_INFO: `/v3/projects/6a31ec5d4f7f4af2abb2cfd0848d26ff/users/dd83be06650843e29927eba212e5140a/roles/a6e562cdc96c4be691d638990f8403fd` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:04.886108 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1f96b15-fbe8-4c5f-8432-2ce952b87a30 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6a31ec5d4f7f4af2abb2cfd0848d26ff, user_id=dd83be06650843e29927eba212e5140a, role_id=a6e562cdc96c4be691d638990f8403fd)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:04.890846 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1f96b15-fbe8-4c5f-8432-2ce952b87a30 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:04.924207 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1047/2095] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:35:04 2022] PUT /identity/v3/projects/6a31ec5d4f7f4af2abb2cfd0848d26ff/users/dd83be06650843e29927eba212e5140a/roles/a6e562cdc96c4be691d638990f8403fd => generated 0 bytes in 109 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 21:35:04.929719 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29c702b5-51a3-4e1d-aba4-e41236cdb335 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:04.946859 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29c702b5-51a3-4e1d-aba4-e41236cdb335 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:05.111199 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cfeec25-f3fd-4426-a749-00debcc78ef4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:05.112034 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cfeec25-f3fd-4426-a749-00debcc78ef4 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:05.112410 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cfeec25-f3fd-4426-a749-00debcc78ef4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:05.112973 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cfeec25-f3fd-4426-a749-00debcc78ef4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:05.114524 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cfeec25-f3fd-4426-a749-00debcc78ef4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:05.124109 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cfeec25-f3fd-4426-a749-00debcc78ef4 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:05.155280 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1049/2096] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:35:04 2022] GET /identity/v3/roles => generated 1420 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:35:05.162868 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1f96b15-fbe8-4c5f-8432-2ce952b87a30 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:05.183848 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1f96b15-fbe8-4c5f-8432-2ce952b87a30 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:05.192814 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bf26acd-c7ad-4708-a12d-293f9a6a1761 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:05.194473 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bf26acd-c7ad-4708-a12d-293f9a6a1761 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:05.194940 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bf26acd-c7ad-4708-a12d-293f9a6a1761 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:05.195804 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bf26acd-c7ad-4708-a12d-293f9a6a1761 admin admin] PATH_INFO: `/v3/projects/6a31ec5d4f7f4af2abb2cfd0848d26ff/users/dd83be06650843e29927eba212e5140a/roles/a6e562cdc96c4be691d638990f8403fd` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:05.199927 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bf26acd-c7ad-4708-a12d-293f9a6a1761 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6a31ec5d4f7f4af2abb2cfd0848d26ff, user_id=dd83be06650843e29927eba212e5140a, role_id=a6e562cdc96c4be691d638990f8403fd)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:05.204941 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bf26acd-c7ad-4708-a12d-293f9a6a1761 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:05.220534 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1048/2097] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:35:05 2022] PUT /identity/v3/projects/6a31ec5d4f7f4af2abb2cfd0848d26ff/users/dd83be06650843e29927eba212e5140a/roles/a6e562cdc96c4be691d638990f8403fd => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 21:35:05.237738 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d3be8c0-2132-409b-8b9a-7d2ec5745e0c None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:05.241779 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d3be8c0-2132-409b-8b9a-7d2ec5745e0c None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:05.241779 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d3be8c0-2132-409b-8b9a-7d2ec5745e0c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:05.287803 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-1d3be8c0-2132-409b-8b9a-7d2ec5745e0c None None] MFA Rules not processed for user `dd83be06650843e29927eba212e5140a`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:35:05.289145 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-1d3be8c0-2132-409b-8b9a-7d2ec5745e0c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:05.411442 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1050/2098] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:35:05 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:35:05.430319 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bf26acd-c7ad-4708-a12d-293f9a6a1761 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:05.433000 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-6bf26acd-c7ad-4708-a12d-293f9a6a1761 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:05.451621 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:35:05.564449 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bf26acd-c7ad-4708-a12d-293f9a6a1761 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:05.585459 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b99f3be-c602-41f8-985d-4b45a6a7184e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dc226f7ddcb49ecacb51ce57fdfc11b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:05.586093 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b99f3be-c602-41f8-985d-4b45a6a7184e service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:05.586466 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b99f3be-c602-41f8-985d-4b45a6a7184e service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:05.587162 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b99f3be-c602-41f8-985d-4b45a6a7184e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:05.717300 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b99f3be-c602-41f8-985d-4b45a6a7184e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:05.719109 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b99f3be-c602-41f8-985d-4b45a6a7184e service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:05.746734 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1049/2099] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:35:05 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 318 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:35:06.727466 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ffcb94e-c3a6-4e4b-89c4-24ae589247a0 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:06.729121 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ffcb94e-c3a6-4e4b-89c4-24ae589247a0 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:06.730326 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ffcb94e-c3a6-4e4b-89c4-24ae589247a0 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:06.732925 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1051/2100] 10.222.0.90 () {54 vars in 921 bytes} [Mon Aug 29 21:35:06 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:35:06.741458 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6464d9f5-f77b-4c64-b401-888724c0da46 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:06.742456 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6464d9f5-f77b-4c64-b401-888724c0da46 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:06.743077 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6464d9f5-f77b-4c64-b401-888724c0da46 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:06.775574 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-6464d9f5-f77b-4c64-b401-888724c0da46 None None] MFA Rules not processed for user `b95fa07a7c0549dbaac49406e2639e21`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:35:06.777224 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-6464d9f5-f77b-4c64-b401-888724c0da46 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:06.818507 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ffcb94e-c3a6-4e4b-89c4-24ae589247a0 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:06.821596 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-3ffcb94e-c3a6-4e4b-89c4-24ae589247a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:06.891908 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1050/2101] 10.222.0.90 () {58 vars in 1018 bytes} [Mon Aug 29 21:35:06 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:35:06.908669 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6464d9f5-f77b-4c64-b401-888724c0da46 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:06.911013 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-6464d9f5-f77b-4c64-b401-888724c0da46 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:06.912913 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ffcb94e-c3a6-4e4b-89c4-24ae589247a0 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:06.917660 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86744a04-345e-4802-9f7f-c9a4adb03416 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:06.918341 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86744a04-345e-4802-9f7f-c9a4adb03416 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:06.918778 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86744a04-345e-4802-9f7f-c9a4adb03416 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:06.919214 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86744a04-345e-4802-9f7f-c9a4adb03416 admin admin] PATH_INFO: `/v3/users/96d3a0e28dc247b5a7ee9434a4d6554c` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:06.920424 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86744a04-345e-4802-9f7f-c9a4adb03416 admin admin] RBAC: Authorizing `identity:delete_user(user_id=96d3a0e28dc247b5a7ee9434a4d6554c)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:06.922427 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86744a04-345e-4802-9f7f-c9a4adb03416 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:06.930492 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6464d9f5-f77b-4c64-b401-888724c0da46 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:06.939182 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a60b797-ee7d-4d29-92f5-e1ccf036d6c3 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b95fa07a7c0549dbaac49406e2639e21', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:06.940172 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a60b797-ee7d-4d29-92f5-e1ccf036d6c3 service placement] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:06.940698 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a60b797-ee7d-4d29-92f5-e1ccf036d6c3 service placement] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:06.941331 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a60b797-ee7d-4d29-92f5-e1ccf036d6c3 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:06.961235 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a60b797-ee7d-4d29-92f5-e1ccf036d6c3 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:06.964708 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a60b797-ee7d-4d29-92f5-e1ccf036d6c3 service placement] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:06.969811 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=96,keystone:DELETE=4 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:35:06.986894 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1051/2102] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:35:06 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 79 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:35:07.025274 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-86744a04-345e-4802-9f7f-c9a4adb03416 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '96d3a0e28dc247b5a7ee9434a4d6554c'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:07.038826 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-86744a04-345e-4802-9f7f-c9a4adb03416 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '96d3a0e28dc247b5a7ee9434a4d6554c'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:07.053134 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-86744a04-345e-4802-9f7f-c9a4adb03416 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '96d3a0e28dc247b5a7ee9434a4d6554c'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:07.054066 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-86744a04-345e-4802-9f7f-c9a4adb03416 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '96d3a0e28dc247b5a7ee9434a4d6554c'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:07.055022 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-86744a04-345e-4802-9f7f-c9a4adb03416 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '96d3a0e28dc247b5a7ee9434a4d6554c'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:07.079213 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1052/2103] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:35:06 2022] DELETE /identity/v3/users/96d3a0e28dc247b5a7ee9434a4d6554c => generated 0 bytes in 261 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:35:07.383857 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a60b797-ee7d-4d29-92f5-e1ccf036d6c3 service placement] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:07.443546 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a60b797-ee7d-4d29-92f5-e1ccf036d6c3 service placement] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:07.557048 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:35:07.592663 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21ad8407-8a41-4d98-957c-07ec97871d37 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:07.593630 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21ad8407-8a41-4d98-957c-07ec97871d37 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:07.594226 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21ad8407-8a41-4d98-957c-07ec97871d37 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:07.594775 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21ad8407-8a41-4d98-957c-07ec97871d37 admin admin] PATH_INFO: `/v3/projects/71f4912b8f1b4a5199fe6d11315c2658` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:07.596701 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21ad8407-8a41-4d98-957c-07ec97871d37 admin admin] RBAC: Authorizing `identity:delete_project(project_id=71f4912b8f1b4a5199fe6d11315c2658)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:07.600860 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21ad8407-8a41-4d98-957c-07ec97871d37 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:07.715515 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-21ad8407-8a41-4d98-957c-07ec97871d37 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '71f4912b8f1b4a5199fe6d11315c2658', 'request_id': 'req-21ad8407-8a41-4d98-957c-07ec97871d37'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:07.726559 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-21ad8407-8a41-4d98-957c-07ec97871d37 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '71f4912b8f1b4a5199fe6d11315c2658', 'request_id': 'req-21ad8407-8a41-4d98-957c-07ec97871d37'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:07.740936 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-21ad8407-8a41-4d98-957c-07ec97871d37 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '71f4912b8f1b4a5199fe6d11315c2658', 'request_id': 'req-21ad8407-8a41-4d98-957c-07ec97871d37'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:07.766998 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-21ad8407-8a41-4d98-957c-07ec97871d37 admin admin] The token cache is being invalidate because project 71f4912b8f1b4a5199fe6d11315c2658 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:35:07.768691 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-21ad8407-8a41-4d98-957c-07ec97871d37 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:07.771080 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-21ad8407-8a41-4d98-957c-07ec97871d37 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:07.775048 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1052/2104] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:35:07 2022] DELETE /identity/v3/projects/71f4912b8f1b4a5199fe6d11315c2658 => generated 0 bytes in 392 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:35:07.781424 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4d0bb93-72b5-4c41-81a5-5ded9dbe98ef None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:07.782234 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4d0bb93-72b5-4c41-81a5-5ded9dbe98ef None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:07.782519 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4d0bb93-72b5-4c41-81a5-5ded9dbe98ef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:07.875672 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-b4d0bb93-72b5-4c41-81a5-5ded9dbe98ef None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:35:07.877768 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-b4d0bb93-72b5-4c41-81a5-5ded9dbe98ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:07.883978 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21ad8407-8a41-4d98-957c-07ec97871d37 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:07.891099 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-21ad8407-8a41-4d98-957c-07ec97871d37 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:08.102154 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21ad8407-8a41-4d98-957c-07ec97871d37 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:08.108807 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e31fff6c-d774-4ce8-92b8-79dcc0b52116 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dc226f7ddcb49ecacb51ce57fdfc11b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:08.109728 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e31fff6c-d774-4ce8-92b8-79dcc0b52116 service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:08.110141 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e31fff6c-d774-4ce8-92b8-79dcc0b52116 service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:08.110581 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e31fff6c-d774-4ce8-92b8-79dcc0b52116 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:08.115303 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-e31fff6c-d774-4ce8-92b8-79dcc0b52116 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:08.134681 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1053/2105] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:35:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 355 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 21:35:08.155069 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27164665-f489-4f7e-acd4-0f4b67b5d617 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:08.156601 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27164665-f489-4f7e-acd4-0f4b67b5d617 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:08.157460 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27164665-f489-4f7e-acd4-0f4b67b5d617 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:08.212437 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-27164665-f489-4f7e-acd4-0f4b67b5d617 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:35:08.215343 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-27164665-f489-4f7e-acd4-0f4b67b5d617 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:08.232033 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1054/2106] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:35:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:35:08.232912 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27164665-f489-4f7e-acd4-0f4b67b5d617 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:08.234919 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-27164665-f489-4f7e-acd4-0f4b67b5d617 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:08.255657 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27164665-f489-4f7e-acd4-0f4b67b5d617 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:08.269565 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d956a217-1e46-48bb-97a8-199a12bb57ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:08.270733 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d956a217-1e46-48bb-97a8-199a12bb57ab admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:08.271574 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d956a217-1e46-48bb-97a8-199a12bb57ab admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:08.272022 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d956a217-1e46-48bb-97a8-199a12bb57ab admin admin] PATH_INFO: `/v3/users/2444cf2966d94c3f9cf61774ce88e48d` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:08.273254 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e31fff6c-d774-4ce8-92b8-79dcc0b52116 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:08.274326 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d956a217-1e46-48bb-97a8-199a12bb57ab admin admin] RBAC: Authorizing `identity:delete_user(user_id=2444cf2966d94c3f9cf61774ce88e48d)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:08.275261 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e31fff6c-d774-4ce8-92b8-79dcc0b52116 service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:08.278231 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d956a217-1e46-48bb-97a8-199a12bb57ab admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:08.291783 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1053/2107] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:35:07 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 409 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 21:35:08.295739 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e31fff6c-d774-4ce8-92b8-79dcc0b52116 service cinder] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:08.314522 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e31fff6c-d774-4ce8-92b8-79dcc0b52116 service cinder] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:08.323810 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60bfe6fb-8700-4eba-a488-33295d81dd8e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:08.324917 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60bfe6fb-8700-4eba-a488-33295d81dd8e admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:08.325696 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60bfe6fb-8700-4eba-a488-33295d81dd8e admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:08.326202 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60bfe6fb-8700-4eba-a488-33295d81dd8e admin admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:08.327587 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60bfe6fb-8700-4eba-a488-33295d81dd8e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:35:08.328029 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60bfe6fb-8700-4eba-a488-33295d81dd8e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:08.331281 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60bfe6fb-8700-4eba-a488-33295d81dd8e admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:08.357844 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1054/2108] 10.222.0.90 () {58 vars in 1153 bytes} [Mon Aug 29 21:35:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 61 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:35:08.360840 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60bfe6fb-8700-4eba-a488-33295d81dd8e admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:08.362503 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-d956a217-1e46-48bb-97a8-199a12bb57ab admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2444cf2966d94c3f9cf61774ce88e48d'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:08.379756 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-d956a217-1e46-48bb-97a8-199a12bb57ab admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2444cf2966d94c3f9cf61774ce88e48d'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:08.386430 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60bfe6fb-8700-4eba-a488-33295d81dd8e admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:08.389013 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-d956a217-1e46-48bb-97a8-199a12bb57ab admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2444cf2966d94c3f9cf61774ce88e48d'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:08.389990 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-d956a217-1e46-48bb-97a8-199a12bb57ab admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2444cf2966d94c3f9cf61774ce88e48d'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:08.390745 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-d956a217-1e46-48bb-97a8-199a12bb57ab admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2444cf2966d94c3f9cf61774ce88e48d'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:08.392507 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed28c5a0-fc8c-4ed8-a3dd-7f328c522c07 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:08.393153 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed28c5a0-fc8c-4ed8-a3dd-7f328c522c07 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:08.393413 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed28c5a0-fc8c-4ed8-a3dd-7f328c522c07 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:08.393688 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed28c5a0-fc8c-4ed8-a3dd-7f328c522c07 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:08.395045 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed28c5a0-fc8c-4ed8-a3dd-7f328c522c07 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:08.397451 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed28c5a0-fc8c-4ed8-a3dd-7f328c522c07 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:08.419937 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:35:08.427921 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1055/2109] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:35:08 2022] DELETE /identity/v3/users/2444cf2966d94c3f9cf61774ce88e48d => generated 0 bytes in 195 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:35:08.435254 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42a78524-9c3c-4280-96e5-b85c743f7f34 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:08.435618 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42a78524-9c3c-4280-96e5-b85c743f7f34 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:08.436355 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42a78524-9c3c-4280-96e5-b85c743f7f34 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:08.445910 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1055/2110] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:35:08 2022] POST /identity/v3/projects => generated 387 bytes in 86 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 21:35:08.451670 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed28c5a0-fc8c-4ed8-a3dd-7f328c522c07 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:08.457539 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-ed28c5a0-fc8c-4ed8-a3dd-7f328c522c07 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:08.460515 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=82,keystone:DELETE=16,keystone:INSERT=2 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:35:08.476145 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-42a78524-9c3c-4280-96e5-b85c743f7f34 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:35:08.478489 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-42a78524-9c3c-4280-96e5-b85c743f7f34 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:08.602195 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1056/2111] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:35:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:35:08.612404 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42a78524-9c3c-4280-96e5-b85c743f7f34 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:08.612831 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed28c5a0-fc8c-4ed8-a3dd-7f328c522c07 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:08.615687 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-42a78524-9c3c-4280-96e5-b85c743f7f34 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:08.621032 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4342b18b-ae9e-4387-a2fc-0379ec9a5aa2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:08.622287 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4342b18b-ae9e-4387-a2fc-0379ec9a5aa2 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:08.622740 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4342b18b-ae9e-4387-a2fc-0379ec9a5aa2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:08.623186 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4342b18b-ae9e-4387-a2fc-0379ec9a5aa2 admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:08.632375 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4342b18b-ae9e-4387-a2fc-0379ec9a5aa2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:08.632375 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4342b18b-ae9e-4387-a2fc-0379ec9a5aa2 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:08.690091 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1056/2112] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:35:08 2022] POST /identity/v3/users => generated 294 bytes in 239 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 21:35:08.697743 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4342b18b-ae9e-4387-a2fc-0379ec9a5aa2 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:08.730266 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4342b18b-ae9e-4387-a2fc-0379ec9a5aa2 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:08.740048 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b483df8-799f-46dd-8aa2-d2ecee6c4e3b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:08.741253 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b483df8-799f-46dd-8aa2-d2ecee6c4e3b admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:08.741948 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b483df8-799f-46dd-8aa2-d2ecee6c4e3b admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:08.742655 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b483df8-799f-46dd-8aa2-d2ecee6c4e3b admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:08.744014 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b483df8-799f-46dd-8aa2-d2ecee6c4e3b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:08.747878 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b483df8-799f-46dd-8aa2-d2ecee6c4e3b admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:08.758369 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42a78524-9c3c-4280-96e5-b85c743f7f34 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:08.774316 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de61e311-3b98-4e3a-b73c-d41a4a90857f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:08.775412 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de61e311-3b98-4e3a-b73c-d41a4a90857f service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:08.775954 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de61e311-3b98-4e3a-b73c-d41a4a90857f service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:08.776425 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de61e311-3b98-4e3a-b73c-d41a4a90857f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:08.783747 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1057/2113] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:35:08 2022] GET /identity/v3/roles => generated 1420 bytes in 89 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 29 21:35:08.790760 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b483df8-799f-46dd-8aa2-d2ecee6c4e3b admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:08.802668 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de61e311-3b98-4e3a-b73c-d41a4a90857f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:08.804149 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de61e311-3b98-4e3a-b73c-d41a4a90857f service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:08.804691 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b483df8-799f-46dd-8aa2-d2ecee6c4e3b admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:08.816547 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fee9155-f4e5-4e8b-9a18-62fa5b83f885 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:08.817188 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fee9155-f4e5-4e8b-9a18-62fa5b83f885 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:08.817475 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fee9155-f4e5-4e8b-9a18-62fa5b83f885 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:08.817755 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fee9155-f4e5-4e8b-9a18-62fa5b83f885 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:08.818462 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fee9155-f4e5-4e8b-9a18-62fa5b83f885 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:08.820131 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fee9155-f4e5-4e8b-9a18-62fa5b83f885 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:08.822193 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1057/2114] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:35:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 212 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 21:35:08.866114 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1058/2115] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:35:08 2022] GET /identity/v3/roles => generated 1420 bytes in 76 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:35:08.870839 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de61e311-3b98-4e3a-b73c-d41a4a90857f service neutron] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:08.885079 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de61e311-3b98-4e3a-b73c-d41a4a90857f service neutron] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:08.892605 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bc7213a-b9f4-47f9-93f2-5d57517b34fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:08.893615 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bc7213a-b9f4-47f9-93f2-5d57517b34fe admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:08.894538 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bc7213a-b9f4-47f9-93f2-5d57517b34fe admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:08.895633 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bc7213a-b9f4-47f9-93f2-5d57517b34fe admin admin] PATH_INFO: `/v3/projects/817ab3905c8541dc9c2bab51764984d0/users/b6a3df8d796e4ae6ac358f103903d427/roles/02e242d5520740d4b77a5f4008c7eb57` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:08.918034 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bc7213a-b9f4-47f9-93f2-5d57517b34fe admin admin] RBAC: Authorizing `identity:create_grant(project_id=817ab3905c8541dc9c2bab51764984d0, user_id=b6a3df8d796e4ae6ac358f103903d427, role_id=02e242d5520740d4b77a5f4008c7eb57)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:08.922609 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bc7213a-b9f4-47f9-93f2-5d57517b34fe admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:08.944200 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1058/2116] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:35:08 2022] PUT /identity/v3/projects/817ab3905c8541dc9c2bab51764984d0/users/b6a3df8d796e4ae6ac358f103903d427/roles/02e242d5520740d4b77a5f4008c7eb57 => generated 0 bytes in 74 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 21:35:08.952517 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56b78d5b-f463-4c06-96ba-34f93d0a4640 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:08.953494 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56b78d5b-f463-4c06-96ba-34f93d0a4640 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:08.953989 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56b78d5b-f463-4c06-96ba-34f93d0a4640 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:09.007440 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-56b78d5b-f463-4c06-96ba-34f93d0a4640 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:35:09.009571 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-56b78d5b-f463-4c06-96ba-34f93d0a4640 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:09.258329 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1059/2117] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:35:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 308 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 21:35:09.275909 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bc7213a-b9f4-47f9-93f2-5d57517b34fe admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:09.279533 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-6bc7213a-b9f4-47f9-93f2-5d57517b34fe admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:09.385818 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56b78d5b-f463-4c06-96ba-34f93d0a4640 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:09.388276 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-56b78d5b-f463-4c06-96ba-34f93d0a4640 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:09.452439 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bc7213a-b9f4-47f9-93f2-5d57517b34fe admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:09.458083 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5105176f-a99b-4cda-8d37-0af40610394f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:09.459276 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5105176f-a99b-4cda-8d37-0af40610394f service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:09.459734 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5105176f-a99b-4cda-8d37-0af40610394f service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:09.460131 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5105176f-a99b-4cda-8d37-0af40610394f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:09.467850 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56b78d5b-f463-4c06-96ba-34f93d0a4640 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:09.474353 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5105176f-a99b-4cda-8d37-0af40610394f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:09.476086 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5105176f-a99b-4cda-8d37-0af40610394f service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:09.483711 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfc1750e-3f7e-4ccd-8ae1-a44ccdae09c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:09.484811 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfc1750e-3f7e-4ccd-8ae1-a44ccdae09c5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:09.485296 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfc1750e-3f7e-4ccd-8ae1-a44ccdae09c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:09.486045 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfc1750e-3f7e-4ccd-8ae1-a44ccdae09c5 admin admin] PATH_INFO: `/v3/projects/c54de3a6284c4742b8c30e0021f71f00` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:09.489699 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfc1750e-3f7e-4ccd-8ae1-a44ccdae09c5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c54de3a6284c4742b8c30e0021f71f00)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:09.492513 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1059/2118] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:35:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 218 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 21:35:09.496271 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfc1750e-3f7e-4ccd-8ae1-a44ccdae09c5 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:09.522576 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5105176f-a99b-4cda-8d37-0af40610394f service neutron] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:09.524748 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-5105176f-a99b-4cda-8d37-0af40610394f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:09.554487 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5105176f-a99b-4cda-8d37-0af40610394f service neutron] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:09.588667 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-dfc1750e-3f7e-4ccd-8ae1-a44ccdae09c5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c54de3a6284c4742b8c30e0021f71f00', 'request_id': 'req-dfc1750e-3f7e-4ccd-8ae1-a44ccdae09c5'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:09.607502 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-dfc1750e-3f7e-4ccd-8ae1-a44ccdae09c5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c54de3a6284c4742b8c30e0021f71f00', 'request_id': 'req-dfc1750e-3f7e-4ccd-8ae1-a44ccdae09c5'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:09.612188 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:INSERT=4,keystone:SELECT=91,keystone:DELETE=5 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:35:09.617819 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:35:09.623487 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-dfc1750e-3f7e-4ccd-8ae1-a44ccdae09c5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c54de3a6284c4742b8c30e0021f71f00', 'request_id': 'req-dfc1750e-3f7e-4ccd-8ae1-a44ccdae09c5'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:09.654038 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-dfc1750e-3f7e-4ccd-8ae1-a44ccdae09c5 admin admin] The token cache is being invalidate because project c54de3a6284c4742b8c30e0021f71f00 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:35:09.655089 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-dfc1750e-3f7e-4ccd-8ae1-a44ccdae09c5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:09.656296 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-dfc1750e-3f7e-4ccd-8ae1-a44ccdae09c5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:09.660919 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1060/2119] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:35:09 2022] DELETE /identity/v3/projects/c54de3a6284c4742b8c30e0021f71f00 => generated 0 bytes in 277 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:35:09.667612 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e255e294-6b3c-4623-889f-8938f2e105fd None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:09.667904 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e255e294-6b3c-4623-889f-8938f2e105fd None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:09.668183 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e255e294-6b3c-4623-889f-8938f2e105fd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:09.719955 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-e255e294-6b3c-4623-889f-8938f2e105fd None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:35:09.721675 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-e255e294-6b3c-4623-889f-8938f2e105fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:09.723598 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac014f30-2157-4979-9d15-aae22066a1d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:09.724548 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac014f30-2157-4979-9d15-aae22066a1d4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:09.725135 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac014f30-2157-4979-9d15-aae22066a1d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:09.725539 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac014f30-2157-4979-9d15-aae22066a1d4 admin admin] PATH_INFO: `/v3/users/e818fd39b54f41078d3c00bdb547ff9f` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:09.728234 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac014f30-2157-4979-9d15-aae22066a1d4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e818fd39b54f41078d3c00bdb547ff9f)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:09.731893 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac014f30-2157-4979-9d15-aae22066a1d4 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:09.742618 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1061/2120] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:35:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 76 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:35:09.746201 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e255e294-6b3c-4623-889f-8938f2e105fd None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:09.748016 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-e255e294-6b3c-4623-889f-8938f2e105fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:09.876028 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e255e294-6b3c-4623-889f-8938f2e105fd None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:09.882611 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-ac014f30-2157-4979-9d15-aae22066a1d4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e818fd39b54f41078d3c00bdb547ff9f'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:09.884509 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f93bfe0-9881-4079-bf3b-4bec88346962 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dc226f7ddcb49ecacb51ce57fdfc11b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:09.885073 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f93bfe0-9881-4079-bf3b-4bec88346962 service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:09.885423 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f93bfe0-9881-4079-bf3b-4bec88346962 service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:09.885708 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f93bfe0-9881-4079-bf3b-4bec88346962 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:09.888182 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-3f93bfe0-9881-4079-bf3b-4bec88346962 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:09.900882 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-ac014f30-2157-4979-9d15-aae22066a1d4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e818fd39b54f41078d3c00bdb547ff9f'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:09.920798 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-ac014f30-2157-4979-9d15-aae22066a1d4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e818fd39b54f41078d3c00bdb547ff9f'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:09.922917 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-ac014f30-2157-4979-9d15-aae22066a1d4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e818fd39b54f41078d3c00bdb547ff9f'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:09.923765 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-ac014f30-2157-4979-9d15-aae22066a1d4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e818fd39b54f41078d3c00bdb547ff9f'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:09.973643 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1060/2121] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:35:09 2022] DELETE /identity/v3/users/e818fd39b54f41078d3c00bdb547ff9f => generated 0 bytes in 452 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:35:09.976435 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04d0b125-9e5b-43fa-bad9-9bbcd39e41fd None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:09.976723 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04d0b125-9e5b-43fa-bad9-9bbcd39e41fd None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:09.977372 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04d0b125-9e5b-43fa-bad9-9bbcd39e41fd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:10.011897 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-04d0b125-9e5b-43fa-bad9-9bbcd39e41fd None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:35:10.014307 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-04d0b125-9e5b-43fa-bad9-9bbcd39e41fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:10.052660 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f93bfe0-9881-4079-bf3b-4bec88346962 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:10.055094 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f93bfe0-9881-4079-bf3b-4bec88346962 service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:10.068882 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1062/2122] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:35:09 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 324 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:35:10.070514 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04da0068-98e3-4e9d-9a63-7f22227cf541 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:10.070809 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04da0068-98e3-4e9d-9a63-7f22227cf541 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:10.071155 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04da0068-98e3-4e9d-9a63-7f22227cf541 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:10.103926 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-04da0068-98e3-4e9d-9a63-7f22227cf541 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:35:10.106020 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-04da0068-98e3-4e9d-9a63-7f22227cf541 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:10.130564 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1061/2123] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:35:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:35:10.138931 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04d0b125-9e5b-43fa-bad9-9bbcd39e41fd None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:10.154601 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04d0b125-9e5b-43fa-bad9-9bbcd39e41fd None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:10.183195 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66928839-d3f0-46aa-8a01-b16d2a70fc11 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:10.183445 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66928839-d3f0-46aa-8a01-b16d2a70fc11 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:10.186678 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66928839-d3f0-46aa-8a01-b16d2a70fc11 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:10.187229 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66928839-d3f0-46aa-8a01-b16d2a70fc11 admin admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:10.188530 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66928839-d3f0-46aa-8a01-b16d2a70fc11 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:35:10.189050 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66928839-d3f0-46aa-8a01-b16d2a70fc11 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:10.202185 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66928839-d3f0-46aa-8a01-b16d2a70fc11 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:10.256213 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1063/2124] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:35:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:35:10.257963 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1062/2125] 10.222.0.90 () {58 vars in 1153 bytes} [Mon Aug 29 21:35:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 123 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:35:10.264724 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04da0068-98e3-4e9d-9a63-7f22227cf541 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:10.280873 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04da0068-98e3-4e9d-9a63-7f22227cf541 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:10.288012 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66928839-d3f0-46aa-8a01-b16d2a70fc11 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:10.289941 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-66928839-d3f0-46aa-8a01-b16d2a70fc11 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:10.290161 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc0c59eb-d568-44a9-9332-95c1a51e2377 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:10.291092 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc0c59eb-d568-44a9-9332-95c1a51e2377 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:10.291763 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc0c59eb-d568-44a9-9332-95c1a51e2377 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:10.292277 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc0c59eb-d568-44a9-9332-95c1a51e2377 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:10.293196 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc0c59eb-d568-44a9-9332-95c1a51e2377 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:10.295610 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc0c59eb-d568-44a9-9332-95c1a51e2377 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:10.357168 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:35:10.362222 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1064/2126] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:35:10 2022] POST /identity/v3/projects => generated 377 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:35:10.369909 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc0c59eb-d568-44a9-9332-95c1a51e2377 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:10.390398 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66928839-d3f0-46aa-8a01-b16d2a70fc11 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:10.395608 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=96,keystone:DELETE=2,keystone:INSERT=2 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:35:10.516529 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3b28109-b4b3-4909-a303-71ddef3704e4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:10.517767 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3b28109-b4b3-4909-a303-71ddef3704e4 service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:10.521330 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3b28109-b4b3-4909-a303-71ddef3704e4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:10.522034 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3b28109-b4b3-4909-a303-71ddef3704e4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:10.523737 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc0c59eb-d568-44a9-9332-95c1a51e2377 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:10.531845 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78aadcf3-07fd-4bec-9c07-d0906673ad47 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:10.532642 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78aadcf3-07fd-4bec-9c07-d0906673ad47 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:10.532927 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78aadcf3-07fd-4bec-9c07-d0906673ad47 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:10.533324 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78aadcf3-07fd-4bec-9c07-d0906673ad47 admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:10.534326 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78aadcf3-07fd-4bec-9c07-d0906673ad47 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:10.543970 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78aadcf3-07fd-4bec-9c07-d0906673ad47 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:10.551929 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3b28109-b4b3-4909-a303-71ddef3704e4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:10.554631 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3b28109-b4b3-4909-a303-71ddef3704e4 service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:10.582514 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1065/2127] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:35:10 2022] POST /identity/v3/users => generated 289 bytes in 214 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 21:35:10.583781 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1063/2128] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:35:10 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 296 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 21:35:10.588008 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78aadcf3-07fd-4bec-9c07-d0906673ad47 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:10.611637 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78aadcf3-07fd-4bec-9c07-d0906673ad47 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:10.618360 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1913267e-bc83-4c25-b798-b27622563dce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:10.619640 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1913267e-bc83-4c25-b798-b27622563dce admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:10.620505 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1913267e-bc83-4c25-b798-b27622563dce admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:10.621583 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1913267e-bc83-4c25-b798-b27622563dce admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:10.622073 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1913267e-bc83-4c25-b798-b27622563dce admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:10.624355 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1913267e-bc83-4c25-b798-b27622563dce admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:10.636279 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1066/2129] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:35:10 2022] GET /identity/v3/roles => generated 1420 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 29 21:35:10.642701 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3b28109-b4b3-4909-a303-71ddef3704e4 service neutron] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:10.655172 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3b28109-b4b3-4909-a303-71ddef3704e4 service neutron] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:10.665773 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd9fce23-1dec-4cdd-9d0b-ec0ede33f516 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:10.666648 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd9fce23-1dec-4cdd-9d0b-ec0ede33f516 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:10.667029 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd9fce23-1dec-4cdd-9d0b-ec0ede33f516 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:10.667411 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd9fce23-1dec-4cdd-9d0b-ec0ede33f516 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:10.668182 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd9fce23-1dec-4cdd-9d0b-ec0ede33f516 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:10.672061 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd9fce23-1dec-4cdd-9d0b-ec0ede33f516 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:10.689892 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1064/2130] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:35:10 2022] GET /identity/v3/roles => generated 1420 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:35:10.696030 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1913267e-bc83-4c25-b798-b27622563dce admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:10.709503 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1913267e-bc83-4c25-b798-b27622563dce admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:10.716166 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ae54b3a-4303-48fb-a589-b0d30e6e4c2a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:10.716868 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae54b3a-4303-48fb-a589-b0d30e6e4c2a admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:10.717219 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae54b3a-4303-48fb-a589-b0d30e6e4c2a admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:10.717705 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae54b3a-4303-48fb-a589-b0d30e6e4c2a admin admin] PATH_INFO: `/v3/projects/44beb31da9fb4270ac4cfcd7fd987fd4/users/b5e69fef981244908ae6f53516e20bdf/roles/02e242d5520740d4b77a5f4008c7eb57` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:10.740094 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ae54b3a-4303-48fb-a589-b0d30e6e4c2a admin admin] RBAC: Authorizing `identity:create_grant(project_id=44beb31da9fb4270ac4cfcd7fd987fd4, user_id=b5e69fef981244908ae6f53516e20bdf, role_id=02e242d5520740d4b77a5f4008c7eb57)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:10.746381 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ae54b3a-4303-48fb-a589-b0d30e6e4c2a admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:10.793917 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1067/2131] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:35:10 2022] PUT /identity/v3/projects/44beb31da9fb4270ac4cfcd7fd987fd4/users/b5e69fef981244908ae6f53516e20bdf/roles/02e242d5520740d4b77a5f4008c7eb57 => generated 0 bytes in 93 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:35:10.809111 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e9f6112-99d6-4dd3-9e35-35299f0fb4fe None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:10.809629 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e9f6112-99d6-4dd3-9e35-35299f0fb4fe None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:10.810086 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e9f6112-99d6-4dd3-9e35-35299f0fb4fe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:10.879984 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-8e9f6112-99d6-4dd3-9e35-35299f0fb4fe None None] MFA Rules not processed for user `b5e69fef981244908ae6f53516e20bdf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:35:10.881990 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-8e9f6112-99d6-4dd3-9e35-35299f0fb4fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:11.111279 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1065/2132] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:35:10 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 301 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:35:11.115276 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ae54b3a-4303-48fb-a589-b0d30e6e4c2a admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:11.262515 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ae54b3a-4303-48fb-a589-b0d30e6e4c2a admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:11.282017 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa69bba4-6c8a-4da0-b7e2-4a9c721d095e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:11.283070 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa69bba4-6c8a-4da0-b7e2-4a9c721d095e admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:11.283567 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa69bba4-6c8a-4da0-b7e2-4a9c721d095e admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:11.284093 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa69bba4-6c8a-4da0-b7e2-4a9c721d095e admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:11.286655 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa69bba4-6c8a-4da0-b7e2-4a9c721d095e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:11.289184 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa69bba4-6c8a-4da0-b7e2-4a9c721d095e admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:11.368705 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1068/2133] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:35:11 2022] POST /identity/v3/projects => generated 377 bytes in 256 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:35:11.372396 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e9f6112-99d6-4dd3-9e35-35299f0fb4fe None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:11.397050 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e9f6112-99d6-4dd3-9e35-35299f0fb4fe None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:11.554048 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5295183b-796b-476f-9f48-01b358908729 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:11.556429 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5295183b-796b-476f-9f48-01b358908729 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:11.557482 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5295183b-796b-476f-9f48-01b358908729 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:11.558589 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5295183b-796b-476f-9f48-01b358908729 admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:11.559477 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5295183b-796b-476f-9f48-01b358908729 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:11.562765 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5295183b-796b-476f-9f48-01b358908729 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:11.618052 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa69bba4-6c8a-4da0-b7e2-4a9c721d095e admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:11.620423 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-aa69bba4-6c8a-4da0-b7e2-4a9c721d095e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:11.634904 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1066/2134] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:35:11 2022] POST /identity/v3/users => generated 289 bytes in 262 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:35:11.640055 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5295183b-796b-476f-9f48-01b358908729 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:11.654993 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:35:11.658210 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa69bba4-6c8a-4da0-b7e2-4a9c721d095e admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:11.673915 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5295183b-796b-476f-9f48-01b358908729 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:11.677948 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53aa6e44-4066-44f0-98d4-26027e0dec5a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:11.679137 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53aa6e44-4066-44f0-98d4-26027e0dec5a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:11.679628 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53aa6e44-4066-44f0-98d4-26027e0dec5a admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:11.680291 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53aa6e44-4066-44f0-98d4-26027e0dec5a admin admin] PATH_INFO: `/v3/projects/604e07baa03b4cb5a0e2c94bfd59993d` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:11.684622 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53aa6e44-4066-44f0-98d4-26027e0dec5a admin admin] RBAC: Authorizing `identity:delete_project(project_id=604e07baa03b4cb5a0e2c94bfd59993d)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:11.687876 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53aa6e44-4066-44f0-98d4-26027e0dec5a admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:11.698121 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5df210d7-e2e3-4612-94b6-4c8563fae96b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:11.699118 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5df210d7-e2e3-4612-94b6-4c8563fae96b admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:11.699582 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5df210d7-e2e3-4612-94b6-4c8563fae96b admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:11.700188 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5df210d7-e2e3-4612-94b6-4c8563fae96b admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:11.704833 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5df210d7-e2e3-4612-94b6-4c8563fae96b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:11.707914 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5df210d7-e2e3-4612-94b6-4c8563fae96b admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:11.747684 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1067/2135] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:35:11 2022] GET /identity/v3/roles => generated 1420 bytes in 108 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 29 21:35:11.755943 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5df210d7-e2e3-4612-94b6-4c8563fae96b admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:11.807407 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5df210d7-e2e3-4612-94b6-4c8563fae96b admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:11.826824 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eea8f001-c2d0-4ef0-9462-796d6b8b3af8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:11.828428 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea8f001-c2d0-4ef0-9462-796d6b8b3af8 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:11.830225 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea8f001-c2d0-4ef0-9462-796d6b8b3af8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:11.830713 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea8f001-c2d0-4ef0-9462-796d6b8b3af8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:11.833275 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eea8f001-c2d0-4ef0-9462-796d6b8b3af8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:11.842538 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=92,keystone:INSERT=5,keystone:DELETE=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:35:11.843866 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eea8f001-c2d0-4ef0-9462-796d6b8b3af8 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:11.857308 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-53aa6e44-4066-44f0-98d4-26027e0dec5a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '604e07baa03b4cb5a0e2c94bfd59993d', 'request_id': 'req-53aa6e44-4066-44f0-98d4-26027e0dec5a'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:11.874621 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-53aa6e44-4066-44f0-98d4-26027e0dec5a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '604e07baa03b4cb5a0e2c94bfd59993d', 'request_id': 'req-53aa6e44-4066-44f0-98d4-26027e0dec5a'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:11.888950 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-53aa6e44-4066-44f0-98d4-26027e0dec5a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '604e07baa03b4cb5a0e2c94bfd59993d', 'request_id': 'req-53aa6e44-4066-44f0-98d4-26027e0dec5a'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:11.914021 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1068/2136] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:35:11 2022] GET /identity/v3/roles => generated 1420 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:35:11.915873 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eea8f001-c2d0-4ef0-9462-796d6b8b3af8 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:11.920915 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-53aa6e44-4066-44f0-98d4-26027e0dec5a admin admin] The token cache is being invalidate because project 604e07baa03b4cb5a0e2c94bfd59993d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:35:11.921627 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-53aa6e44-4066-44f0-98d4-26027e0dec5a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:11.923706 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-53aa6e44-4066-44f0-98d4-26027e0dec5a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:11.927163 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1069/2137] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:35:11 2022] DELETE /identity/v3/projects/604e07baa03b4cb5a0e2c94bfd59993d => generated 0 bytes in 311 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 21:35:11.934947 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53aa6e44-4066-44f0-98d4-26027e0dec5a admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:11.940108 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eea8f001-c2d0-4ef0-9462-796d6b8b3af8 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:11.944529 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-53aa6e44-4066-44f0-98d4-26027e0dec5a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:12.116837 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4310acf7-5506-4bce-ba11-646a2a30db97 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:12.118939 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4310acf7-5506-4bce-ba11-646a2a30db97 admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:12.119651 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4310acf7-5506-4bce-ba11-646a2a30db97 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:12.120351 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4310acf7-5506-4bce-ba11-646a2a30db97 admin admin] PATH_INFO: `/v3/projects/c61d110514dc41648a9c202238496fa7/users/c7aff8ab512d470ebedea4aaa327e0b5/roles/02e242d5520740d4b77a5f4008c7eb57` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:12.165118 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4310acf7-5506-4bce-ba11-646a2a30db97 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c61d110514dc41648a9c202238496fa7, user_id=c7aff8ab512d470ebedea4aaa327e0b5, role_id=02e242d5520740d4b77a5f4008c7eb57)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:12.172426 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4310acf7-5506-4bce-ba11-646a2a30db97 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:12.196183 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53aa6e44-4066-44f0-98d4-26027e0dec5a admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:12.208427 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1069/2138] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:35:11 2022] PUT /identity/v3/projects/c61d110514dc41648a9c202238496fa7/users/c7aff8ab512d470ebedea4aaa327e0b5/roles/02e242d5520740d4b77a5f4008c7eb57 => generated 0 bytes in 293 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 21:35:12.223488 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13cf5df3-78cf-4411-ac79-ff0d8134cedc None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:12.224005 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13cf5df3-78cf-4411-ac79-ff0d8134cedc None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:12.224168 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13cf5df3-78cf-4411-ac79-ff0d8134cedc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:12.272560 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-13cf5df3-78cf-4411-ac79-ff0d8134cedc None None] MFA Rules not processed for user `c7aff8ab512d470ebedea4aaa327e0b5`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:35:12.274583 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-13cf5df3-78cf-4411-ac79-ff0d8134cedc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:12.299849 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4849a7d-d36a-4598-a887-e50c0240533c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:12.300867 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4849a7d-d36a-4598-a887-e50c0240533c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:12.301167 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4849a7d-d36a-4598-a887-e50c0240533c admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:12.301418 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4849a7d-d36a-4598-a887-e50c0240533c admin admin] PATH_INFO: `/v3/users/ead580ae5e18434f9def54e46ea72149` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:12.302891 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4849a7d-d36a-4598-a887-e50c0240533c admin admin] RBAC: Authorizing `identity:delete_user(user_id=ead580ae5e18434f9def54e46ea72149)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:12.305991 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4849a7d-d36a-4598-a887-e50c0240533c admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:12.389250 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-a4849a7d-d36a-4598-a887-e50c0240533c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ead580ae5e18434f9def54e46ea72149'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:12.412026 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-a4849a7d-d36a-4598-a887-e50c0240533c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ead580ae5e18434f9def54e46ea72149'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:12.414018 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-a4849a7d-d36a-4598-a887-e50c0240533c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ead580ae5e18434f9def54e46ea72149'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:12.414634 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-a4849a7d-d36a-4598-a887-e50c0240533c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ead580ae5e18434f9def54e46ea72149'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:12.427519 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-a4849a7d-d36a-4598-a887-e50c0240533c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ead580ae5e18434f9def54e46ea72149'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:12.434915 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:35:12.447124 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1070/2139] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:35:12 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 227 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:35:12.453090 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13cf5df3-78cf-4411-ac79-ff0d8134cedc None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:12.454555 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-13cf5df3-78cf-4411-ac79-ff0d8134cedc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:12.468058 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1070/2140] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:35:11 2022] DELETE /identity/v3/users/ead580ae5e18434f9def54e46ea72149 => generated 0 bytes in 533 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:35:12.503554 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13cf5df3-78cf-4411-ac79-ff0d8134cedc None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:12.619773 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:35:12.637667 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d7f0282-6432-48cd-bd38-19a360efcfb8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:12.638424 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d7f0282-6432-48cd-bd38-19a360efcfb8 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:12.638758 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d7f0282-6432-48cd-bd38-19a360efcfb8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:12.638977 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d7f0282-6432-48cd-bd38-19a360efcfb8 admin admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:12.639969 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d7f0282-6432-48cd-bd38-19a360efcfb8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:12.642891 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d7f0282-6432-48cd-bd38-19a360efcfb8 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:12.778115 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1071/2141] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:35:12 2022] POST /identity/v3/projects => generated 377 bytes in 325 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:35:12.779444 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4849a7d-d36a-4598-a887-e50c0240533c admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:12.808900 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4849a7d-d36a-4598-a887-e50c0240533c admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:12.984084 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e35d688c-fe83-482f-a65b-04eb99777709 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:12.984870 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e35d688c-fe83-482f-a65b-04eb99777709 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:12.985222 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e35d688c-fe83-482f-a65b-04eb99777709 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:12.985568 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e35d688c-fe83-482f-a65b-04eb99777709 admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:12.986767 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e35d688c-fe83-482f-a65b-04eb99777709 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:12.988948 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e35d688c-fe83-482f-a65b-04eb99777709 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:13.041644 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1071/2142] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:35:12 2022] POST /identity/v3/users => generated 295 bytes in 263 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:35:13.049614 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d7f0282-6432-48cd-bd38-19a360efcfb8 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:13.077047 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d7f0282-6432-48cd-bd38-19a360efcfb8 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:13.077480 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bff841a0-9367-49ba-aba9-7fb0e8d707b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:13.078175 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e35d688c-fe83-482f-a65b-04eb99777709 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:13.078724 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff841a0-9367-49ba-aba9-7fb0e8d707b5 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:13.079379 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff841a0-9367-49ba-aba9-7fb0e8d707b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:13.079860 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff841a0-9367-49ba-aba9-7fb0e8d707b5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:13.080234 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-e35d688c-fe83-482f-a65b-04eb99777709 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:13.081178 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bff841a0-9367-49ba-aba9-7fb0e8d707b5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:13.084931 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bff841a0-9367-49ba-aba9-7fb0e8d707b5 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:13.113556 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e35d688c-fe83-482f-a65b-04eb99777709 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:13.116280 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1072/2143] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:35:13 2022] GET /identity/v3/roles => generated 1420 bytes in 68 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 29 21:35:13.122072 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bff841a0-9367-49ba-aba9-7fb0e8d707b5 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:13.137313 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14fe4ded-3434-45de-b1f0-b6f913eb5ab0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:13.138966 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14fe4ded-3434-45de-b1f0-b6f913eb5ab0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:13.146980 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14fe4ded-3434-45de-b1f0-b6f913eb5ab0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:13.147470 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14fe4ded-3434-45de-b1f0-b6f913eb5ab0 admin admin] PATH_INFO: `/v3/projects/8263c43ce6824c8b8b9244ee66ff5ae3` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:13.150201 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14fe4ded-3434-45de-b1f0-b6f913eb5ab0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8263c43ce6824c8b8b9244ee66ff5ae3)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:13.153073 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14fe4ded-3434-45de-b1f0-b6f913eb5ab0 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:13.161916 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bff841a0-9367-49ba-aba9-7fb0e8d707b5 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:13.179405 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a17b0622-87a5-400d-8d6f-3f0a3a3c8db8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:13.180081 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a17b0622-87a5-400d-8d6f-3f0a3a3c8db8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:13.180647 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a17b0622-87a5-400d-8d6f-3f0a3a3c8db8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:13.181954 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a17b0622-87a5-400d-8d6f-3f0a3a3c8db8 admin admin] PATH_INFO: `/v3/projects/84397e7a5d0745be8ee33ad50558cafa/users/f2f5aa779f1d46fcb2ac046a214992dc/roles/a6e562cdc96c4be691d638990f8403fd` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:13.220813 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a17b0622-87a5-400d-8d6f-3f0a3a3c8db8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=84397e7a5d0745be8ee33ad50558cafa, user_id=f2f5aa779f1d46fcb2ac046a214992dc, role_id=a6e562cdc96c4be691d638990f8403fd)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:13.223496 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a17b0622-87a5-400d-8d6f-3f0a3a3c8db8 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:13.248425 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1073/2144] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:35:13 2022] PUT /identity/v3/projects/84397e7a5d0745be8ee33ad50558cafa/users/f2f5aa779f1d46fcb2ac046a214992dc/roles/a6e562cdc96c4be691d638990f8403fd => generated 0 bytes in 128 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:35:13.257127 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a17b0622-87a5-400d-8d6f-3f0a3a3c8db8 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:13.267278 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-14fe4ded-3434-45de-b1f0-b6f913eb5ab0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8263c43ce6824c8b8b9244ee66ff5ae3', 'request_id': 'req-14fe4ded-3434-45de-b1f0-b6f913eb5ab0'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:13.280495 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-14fe4ded-3434-45de-b1f0-b6f913eb5ab0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8263c43ce6824c8b8b9244ee66ff5ae3', 'request_id': 'req-14fe4ded-3434-45de-b1f0-b6f913eb5ab0'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:13.296188 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-14fe4ded-3434-45de-b1f0-b6f913eb5ab0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8263c43ce6824c8b8b9244ee66ff5ae3', 'request_id': 'req-14fe4ded-3434-45de-b1f0-b6f913eb5ab0'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:13.336267 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-14fe4ded-3434-45de-b1f0-b6f913eb5ab0 admin admin] The token cache is being invalidate because project 8263c43ce6824c8b8b9244ee66ff5ae3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:35:13.336643 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-14fe4ded-3434-45de-b1f0-b6f913eb5ab0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:13.338110 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-14fe4ded-3434-45de-b1f0-b6f913eb5ab0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:13.341583 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1072/2145] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:35:13 2022] DELETE /identity/v3/projects/8263c43ce6824c8b8b9244ee66ff5ae3 => generated 0 bytes in 263 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:35:13.347613 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14fe4ded-3434-45de-b1f0-b6f913eb5ab0 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:13.349604 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-14fe4ded-3434-45de-b1f0-b6f913eb5ab0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:13.415642 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a17b0622-87a5-400d-8d6f-3f0a3a3c8db8 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:13.486864 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14fe4ded-3434-45de-b1f0-b6f913eb5ab0 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:13.494288 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dda55b7-d42a-4778-af8d-2d0b75987836 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:13.495675 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dda55b7-d42a-4778-af8d-2d0b75987836 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:13.496269 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dda55b7-d42a-4778-af8d-2d0b75987836 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:13.496693 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dda55b7-d42a-4778-af8d-2d0b75987836 admin admin] PATH_INFO: `/v3/users/5fa6fd38803a4aff89f19939cdb74b27` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:13.498597 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dda55b7-d42a-4778-af8d-2d0b75987836 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5fa6fd38803a4aff89f19939cdb74b27)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:13.500751 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dda55b7-d42a-4778-af8d-2d0b75987836 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:13.519895 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=89,keystone:INSERT=4,keystone:DELETE=7 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:35:13.555039 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4344d3b-80b0-4c9a-83a8-7918e0e3d8c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:13.555993 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4344d3b-80b0-4c9a-83a8-7918e0e3d8c1 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:13.556633 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4344d3b-80b0-4c9a-83a8-7918e0e3d8c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:13.556792 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4344d3b-80b0-4c9a-83a8-7918e0e3d8c1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:13.557918 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4344d3b-80b0-4c9a-83a8-7918e0e3d8c1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:13.561251 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4344d3b-80b0-4c9a-83a8-7918e0e3d8c1 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:13.597722 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1074/2146] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:35:13 2022] GET /identity/v3/roles => generated 1420 bytes in 343 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 29 21:35:13.602455 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4344d3b-80b0-4c9a-83a8-7918e0e3d8c1 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:13.605623 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-a4344d3b-80b0-4c9a-83a8-7918e0e3d8c1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:13.618185 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-1dda55b7-d42a-4778-af8d-2d0b75987836 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5fa6fd38803a4aff89f19939cdb74b27'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:13.637862 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4344d3b-80b0-4c9a-83a8-7918e0e3d8c1 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:13.640549 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-1dda55b7-d42a-4778-af8d-2d0b75987836 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5fa6fd38803a4aff89f19939cdb74b27'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:13.641684 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-1dda55b7-d42a-4778-af8d-2d0b75987836 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5fa6fd38803a4aff89f19939cdb74b27'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:13.643482 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-1dda55b7-d42a-4778-af8d-2d0b75987836 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5fa6fd38803a4aff89f19939cdb74b27'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:13.648334 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06788e45-de86-41a9-936e-a933ba834974 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:13.649381 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06788e45-de86-41a9-936e-a933ba834974 admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:13.650080 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06788e45-de86-41a9-936e-a933ba834974 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:13.650526 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06788e45-de86-41a9-936e-a933ba834974 admin admin] PATH_INFO: `/v3/projects/84397e7a5d0745be8ee33ad50558cafa/users/f2f5aa779f1d46fcb2ac046a214992dc/roles/a6e562cdc96c4be691d638990f8403fd` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:13.655159 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06788e45-de86-41a9-936e-a933ba834974 admin admin] RBAC: Authorizing `identity:create_grant(project_id=84397e7a5d0745be8ee33ad50558cafa, user_id=f2f5aa779f1d46fcb2ac046a214992dc, role_id=a6e562cdc96c4be691d638990f8403fd)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:13.658790 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06788e45-de86-41a9-936e-a933ba834974 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:13.662959 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-1dda55b7-d42a-4778-af8d-2d0b75987836 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5fa6fd38803a4aff89f19939cdb74b27'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:13.681852 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1075/2147] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:35:13 2022] PUT /identity/v3/projects/84397e7a5d0745be8ee33ad50558cafa/users/f2f5aa779f1d46fcb2ac046a214992dc/roles/a6e562cdc96c4be691d638990f8403fd => generated 0 bytes in 78 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 21:35:13.691762 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3fcd617-79fb-441c-bcae-ff4fa99c26e3 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:13.692368 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3fcd617-79fb-441c-bcae-ff4fa99c26e3 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:13.692726 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3fcd617-79fb-441c-bcae-ff4fa99c26e3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:13.697590 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1073/2148] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:35:13 2022] DELETE /identity/v3/users/5fa6fd38803a4aff89f19939cdb74b27 => generated 0 bytes in 351 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:35:13.738731 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-a3fcd617-79fb-441c-bcae-ff4fa99c26e3 None None] MFA Rules not processed for user `f2f5aa779f1d46fcb2ac046a214992dc`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:35:13.740934 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-a3fcd617-79fb-441c-bcae-ff4fa99c26e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:13.778900 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:35:13.928531 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1076/2149] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:35:13 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 234 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:35:13.943375 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dda55b7-d42a-4778-af8d-2d0b75987836 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:13.956124 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-1dda55b7-d42a-4778-af8d-2d0b75987836 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:14.092691 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dda55b7-d42a-4778-af8d-2d0b75987836 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:14.102064 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0e711e7-0179-47cc-9a2f-410e0baf2d5f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dc226f7ddcb49ecacb51ce57fdfc11b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:14.105350 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0e711e7-0179-47cc-9a2f-410e0baf2d5f service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:14.106766 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0e711e7-0179-47cc-9a2f-410e0baf2d5f service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:14.107462 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0e711e7-0179-47cc-9a2f-410e0baf2d5f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:14.111676 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-f0e711e7-0179-47cc-9a2f-410e0baf2d5f service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:14.244249 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0e711e7-0179-47cc-9a2f-410e0baf2d5f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:14.252834 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0e711e7-0179-47cc-9a2f-410e0baf2d5f service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:14.255480 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3fcd617-79fb-441c-bcae-ff4fa99c26e3 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:14.259981 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-a3fcd617-79fb-441c-bcae-ff4fa99c26e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:14.322435 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1074/2150] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:35:13 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 380 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 21:35:14.460541 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3fcd617-79fb-441c-bcae-ff4fa99c26e3 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:14.470435 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-050b58d5-a3ec-4924-9f46-53881a059c27 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:14.471546 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-050b58d5-a3ec-4924-9f46-53881a059c27 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:14.472719 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-050b58d5-a3ec-4924-9f46-53881a059c27 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:14.473606 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-050b58d5-a3ec-4924-9f46-53881a059c27 admin admin] PATH_INFO: `/v3/projects/92d71ebbf1d84444bb39977c4b6b9d09` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:14.476037 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-050b58d5-a3ec-4924-9f46-53881a059c27 admin admin] RBAC: Authorizing `identity:delete_project(project_id=92d71ebbf1d84444bb39977c4b6b9d09)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:14.480200 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-050b58d5-a3ec-4924-9f46-53881a059c27 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:14.664877 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-050b58d5-a3ec-4924-9f46-53881a059c27 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '92d71ebbf1d84444bb39977c4b6b9d09', 'request_id': 'req-050b58d5-a3ec-4924-9f46-53881a059c27'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:14.690452 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-050b58d5-a3ec-4924-9f46-53881a059c27 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '92d71ebbf1d84444bb39977c4b6b9d09', 'request_id': 'req-050b58d5-a3ec-4924-9f46-53881a059c27'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:14.721379 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-050b58d5-a3ec-4924-9f46-53881a059c27 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '92d71ebbf1d84444bb39977c4b6b9d09', 'request_id': 'req-050b58d5-a3ec-4924-9f46-53881a059c27'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:14.766758 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-050b58d5-a3ec-4924-9f46-53881a059c27 admin admin] The token cache is being invalidate because project 92d71ebbf1d84444bb39977c4b6b9d09 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81434) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:35:14.767248 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-050b58d5-a3ec-4924-9f46-53881a059c27 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:14.770241 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-050b58d5-a3ec-4924-9f46-53881a059c27 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:14.774928 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1077/2151] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:35:14 2022] DELETE /identity/v3/projects/92d71ebbf1d84444bb39977c4b6b9d09 => generated 0 bytes in 522 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:35:14.784408 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baf2ce32-6524-4e3a-ad5f-fa40ff9d84ea None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:14.785028 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baf2ce32-6524-4e3a-ad5f-fa40ff9d84ea None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:14.785543 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baf2ce32-6524-4e3a-ad5f-fa40ff9d84ea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:14.874544 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-baf2ce32-6524-4e3a-ad5f-fa40ff9d84ea None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:35:14.875808 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-baf2ce32-6524-4e3a-ad5f-fa40ff9d84ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:14.969092 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:DELETE=10,keystone:SELECT=89,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:35:15.031571 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1075/2152] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:35:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 251 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:35:15.041665 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a577fe69-93ae-4c31-b9cb-3ee594d68c16 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:15.042531 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a577fe69-93ae-4c31-b9cb-3ee594d68c16 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:15.043214 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a577fe69-93ae-4c31-b9cb-3ee594d68c16 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:15.136689 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-a577fe69-93ae-4c31-b9cb-3ee594d68c16 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:35:15.145261 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-a577fe69-93ae-4c31-b9cb-3ee594d68c16 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:15.188291 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1078/2153] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:35:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 21:35:15.193728 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baf2ce32-6524-4e3a-ad5f-fa40ff9d84ea None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:15.261723 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baf2ce32-6524-4e3a-ad5f-fa40ff9d84ea None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:15.286183 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-641c543f-da19-4117-b36b-4e54934b8982 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:15.288651 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-641c543f-da19-4117-b36b-4e54934b8982 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:15.290020 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-641c543f-da19-4117-b36b-4e54934b8982 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:15.292743 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-641c543f-da19-4117-b36b-4e54934b8982 admin admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:15.294103 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-641c543f-da19-4117-b36b-4e54934b8982 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:35:15.294624 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-641c543f-da19-4117-b36b-4e54934b8982 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:15.305852 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-641c543f-da19-4117-b36b-4e54934b8982 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:15.352418 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1076/2154] 10.222.0.90 () {58 vars in 1153 bytes} [Mon Aug 29 21:35:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:35:15.354781 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a577fe69-93ae-4c31-b9cb-3ee594d68c16 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:15.374137 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a577fe69-93ae-4c31-b9cb-3ee594d68c16 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:15.383038 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5af52266-0bce-4a2c-9254-aecf0a10ab7b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:15.384723 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5af52266-0bce-4a2c-9254-aecf0a10ab7b admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:15.385272 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5af52266-0bce-4a2c-9254-aecf0a10ab7b admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:15.385907 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5af52266-0bce-4a2c-9254-aecf0a10ab7b admin admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:15.387509 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5af52266-0bce-4a2c-9254-aecf0a10ab7b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:15.393079 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5af52266-0bce-4a2c-9254-aecf0a10ab7b admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:15.467132 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1079/2155] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:35:15 2022] POST /identity/v3/projects => generated 375 bytes in 113 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:35:15.472701 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-641c543f-da19-4117-b36b-4e54934b8982 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:15.498740 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-641c543f-da19-4117-b36b-4e54934b8982 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:15.734818 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ff920a5-c6fb-4cbb-bbdc-3845f4945fb7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:15.735725 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ff920a5-c6fb-4cbb-bbdc-3845f4945fb7 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:15.736103 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ff920a5-c6fb-4cbb-bbdc-3845f4945fb7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:15.736543 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ff920a5-c6fb-4cbb-bbdc-3845f4945fb7 admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:15.737589 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ff920a5-c6fb-4cbb-bbdc-3845f4945fb7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:15.742241 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ff920a5-c6fb-4cbb-bbdc-3845f4945fb7 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:15.820938 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5af52266-0bce-4a2c-9254-aecf0a10ab7b admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:15.821141 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1077/2156] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:35:15 2022] POST /identity/v3/users => generated 288 bytes in 350 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:35:15.866082 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5af52266-0bce-4a2c-9254-aecf0a10ab7b admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:15.890916 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4ef677d-b099-4753-8aa9-494a662c009d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:15.892173 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4ef677d-b099-4753-8aa9-494a662c009d admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:15.892800 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4ef677d-b099-4753-8aa9-494a662c009d admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:15.893480 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4ef677d-b099-4753-8aa9-494a662c009d admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:15.895289 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4ef677d-b099-4753-8aa9-494a662c009d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:15.901547 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4ef677d-b099-4753-8aa9-494a662c009d admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:15.933268 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1080/2157] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:35:15 2022] GET /identity/v3/roles => generated 1420 bytes in 113 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 29 21:35:15.933893 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ff920a5-c6fb-4cbb-bbdc-3845f4945fb7 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:15.959181 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ff920a5-c6fb-4cbb-bbdc-3845f4945fb7 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:15.979541 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4046287e-d2a0-4041-b447-fc517059a590 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:15.979541 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4046287e-d2a0-4041-b447-fc517059a590 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:15.979541 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4046287e-d2a0-4041-b447-fc517059a590 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:15.979541 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4046287e-d2a0-4041-b447-fc517059a590 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:15.982259 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4046287e-d2a0-4041-b447-fc517059a590 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:15.986593 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4046287e-d2a0-4041-b447-fc517059a590 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:16.032200 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1078/2158] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:35:15 2022] GET /identity/v3/roles => generated 1420 bytes in 99 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:35:16.038575 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4ef677d-b099-4753-8aa9-494a662c009d admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:16.062060 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4ef677d-b099-4753-8aa9-494a662c009d admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:16.089821 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f844bc3-98f0-4f3e-ab53-a23ad8148095 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:16.091018 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f844bc3-98f0-4f3e-ab53-a23ad8148095 admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:16.091534 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f844bc3-98f0-4f3e-ab53-a23ad8148095 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:16.092052 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f844bc3-98f0-4f3e-ab53-a23ad8148095 admin admin] PATH_INFO: `/v3/projects/2dd38d98e8f042b69c60978ea8e7742b/users/29a2a13867e7485590d34bb87b81d6f6/roles/02e242d5520740d4b77a5f4008c7eb57` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:16.103615 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:35:16.153982 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f844bc3-98f0-4f3e-ab53-a23ad8148095 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2dd38d98e8f042b69c60978ea8e7742b, user_id=29a2a13867e7485590d34bb87b81d6f6, role_id=02e242d5520740d4b77a5f4008c7eb57)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:16.164051 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f844bc3-98f0-4f3e-ab53-a23ad8148095 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:16.199677 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1081/2159] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:35:16 2022] PUT /identity/v3/projects/2dd38d98e8f042b69c60978ea8e7742b/users/29a2a13867e7485590d34bb87b81d6f6/roles/02e242d5520740d4b77a5f4008c7eb57 => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:35:16.218399 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca45fda-8754-4d24-bbe1-bff630f57484 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:16.219545 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca45fda-8754-4d24-bbe1-bff630f57484 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:16.219888 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca45fda-8754-4d24-bbe1-bff630f57484 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:16.278098 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-bca45fda-8754-4d24-bbe1-bff630f57484 None None] MFA Rules not processed for user `29a2a13867e7485590d34bb87b81d6f6`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:35:16.279388 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-bca45fda-8754-4d24-bbe1-bff630f57484 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:16.394817 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1079/2160] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:35:16 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:35:16.408713 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f844bc3-98f0-4f3e-ab53-a23ad8148095 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:16.412049 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-6f844bc3-98f0-4f3e-ab53-a23ad8148095 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:16.563503 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f844bc3-98f0-4f3e-ab53-a23ad8148095 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:16.573022 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9f4136c-40b3-4100-99c1-78aafe7bf272 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dc226f7ddcb49ecacb51ce57fdfc11b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:16.574097 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9f4136c-40b3-4100-99c1-78aafe7bf272 service cinder] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:16.574097 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9f4136c-40b3-4100-99c1-78aafe7bf272 service cinder] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:16.576053 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9f4136c-40b3-4100-99c1-78aafe7bf272 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:16.606882 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9f4136c-40b3-4100-99c1-78aafe7bf272 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:16.611426 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9f4136c-40b3-4100-99c1-78aafe7bf272 service cinder] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:16.638459 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1082/2161] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:35:16 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 231 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:35:17.673022 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bca45fda-8754-4d24-bbe1-bff630f57484 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:17.675663 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-bca45fda-8754-4d24-bbe1-bff630f57484 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:17.802336 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:35:17.830605 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9f4136c-40b3-4100-99c1-78aafe7bf272 service cinder] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:17.830605 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9f4136c-40b3-4100-99c1-78aafe7bf272 service cinder] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:17.868361 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35f6dfe2-b42c-4fbf-8005-9308c558423f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dc226f7ddcb49ecacb51ce57fdfc11b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:17.869319 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35f6dfe2-b42c-4fbf-8005-9308c558423f service cinder] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:17.873280 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35f6dfe2-b42c-4fbf-8005-9308c558423f service cinder] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:17.873703 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35f6dfe2-b42c-4fbf-8005-9308c558423f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:17.878789 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-35f6dfe2-b42c-4fbf-8005-9308c558423f service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:17.887805 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bca45fda-8754-4d24-bbe1-bff630f57484 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:17.901362 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9e7279e-0209-4035-a1c7-d9af2a68e113 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:17.902076 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9e7279e-0209-4035-a1c7-d9af2a68e113 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:17.902549 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9e7279e-0209-4035-a1c7-d9af2a68e113 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:17.903032 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9e7279e-0209-4035-a1c7-d9af2a68e113 admin admin] PATH_INFO: `/v3/users/1d27e52b1f1b4cf990b53ac23b15dc6f` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:17.907963 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9e7279e-0209-4035-a1c7-d9af2a68e113 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1d27e52b1f1b4cf990b53ac23b15dc6f)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:17.908478 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9e7279e-0209-4035-a1c7-d9af2a68e113 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:18.007772 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-f9e7279e-0209-4035-a1c7-d9af2a68e113 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1d27e52b1f1b4cf990b53ac23b15dc6f'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:18.022964 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35f6dfe2-b42c-4fbf-8005-9308c558423f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:18.024893 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35f6dfe2-b42c-4fbf-8005-9308c558423f service cinder] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:18.033679 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-f9e7279e-0209-4035-a1c7-d9af2a68e113 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1d27e52b1f1b4cf990b53ac23b15dc6f'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:18.035649 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-f9e7279e-0209-4035-a1c7-d9af2a68e113 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1d27e52b1f1b4cf990b53ac23b15dc6f'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:18.036750 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1083/2162] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:35:17 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 217 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 21:35:18.036883 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-f9e7279e-0209-4035-a1c7-d9af2a68e113 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1d27e52b1f1b4cf990b53ac23b15dc6f'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:18.062032 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-f9e7279e-0209-4035-a1c7-d9af2a68e113 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1d27e52b1f1b4cf990b53ac23b15dc6f'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:18.095106 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1080/2163] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:35:17 2022] DELETE /identity/v3/users/1d27e52b1f1b4cf990b53ac23b15dc6f => generated 0 bytes in 423 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:35:18.104012 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fa19aab-407d-4935-b40d-3a5b3573080c None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:18.104012 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fa19aab-407d-4935-b40d-3a5b3573080c None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:18.104801 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fa19aab-407d-4935-b40d-3a5b3573080c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:18.133312 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-9fa19aab-407d-4935-b40d-3a5b3573080c None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:35:18.135379 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-9fa19aab-407d-4935-b40d-3a5b3573080c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:18.276419 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1084/2164] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:35:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:35:18.289096 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9e7279e-0209-4035-a1c7-d9af2a68e113 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:18.295193 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-f9e7279e-0209-4035-a1c7-d9af2a68e113 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:18.477454 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9e7279e-0209-4035-a1c7-d9af2a68e113 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:18.485831 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4248a31d-cc02-469f-9f0c-466f300efff7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:18.489167 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4248a31d-cc02-469f-9f0c-466f300efff7 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:18.490060 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4248a31d-cc02-469f-9f0c-466f300efff7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:18.491995 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4248a31d-cc02-469f-9f0c-466f300efff7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:18.541879 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4248a31d-cc02-469f-9f0c-466f300efff7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:18.544941 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4248a31d-cc02-469f-9f0c-466f300efff7 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:18.562225 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1081/2165] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:35:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 274 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 21:35:18.642051 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4377a078-7a10-469b-9b09-33442a5e8a8a None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:18.642524 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4377a078-7a10-469b-9b09-33442a5e8a8a None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:18.643166 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4377a078-7a10-469b-9b09-33442a5e8a8a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:18.694253 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-4377a078-7a10-469b-9b09-33442a5e8a8a None None] MFA Rules not processed for user `29a2a13867e7485590d34bb87b81d6f6`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:35:18.704535 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-4377a078-7a10-469b-9b09-33442a5e8a8a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:18.753215 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:35:18.938055 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1085/2166] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:35:18 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 298 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:35:19.002544 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97fdc2e9-8f24-46a4-a8ff-7a1120bb2edc None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:19.003923 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97fdc2e9-8f24-46a4-a8ff-7a1120bb2edc None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:19.004425 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97fdc2e9-8f24-46a4-a8ff-7a1120bb2edc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:19.191858 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-97fdc2e9-8f24-46a4-a8ff-7a1120bb2edc None None] MFA Rules not processed for user `b6a3df8d796e4ae6ac358f103903d427`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:35:19.191858 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-97fdc2e9-8f24-46a4-a8ff-7a1120bb2edc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:19.349208 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:35:19.450172 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1082/2167] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:35:19 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 449 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:35:19.526682 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4377a078-7a10-469b-9b09-33442a5e8a8a None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:19.528454 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-4377a078-7a10-469b-9b09-33442a5e8a8a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:19.645825 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97fdc2e9-8f24-46a4-a8ff-7a1120bb2edc None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:19.647612 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-97fdc2e9-8f24-46a4-a8ff-7a1120bb2edc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:19.661366 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4377a078-7a10-469b-9b09-33442a5e8a8a None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:19.672204 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97fdc2e9-8f24-46a4-a8ff-7a1120bb2edc None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:19.674092 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76dcfe03-348f-49ab-ba3d-f3f9444c5639 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dc226f7ddcb49ecacb51ce57fdfc11b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:19.674765 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76dcfe03-348f-49ab-ba3d-f3f9444c5639 service cinder] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:19.675773 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76dcfe03-348f-49ab-ba3d-f3f9444c5639 service cinder] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:19.676078 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76dcfe03-348f-49ab-ba3d-f3f9444c5639 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:19.685093 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-76dcfe03-348f-49ab-ba3d-f3f9444c5639 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:19.718538 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b441eac-6bcb-49eb-86c3-8e513cdfee8f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:19.718538 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b441eac-6bcb-49eb-86c3-8e513cdfee8f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:19.719078 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b441eac-6bcb-49eb-86c3-8e513cdfee8f admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:19.719078 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b441eac-6bcb-49eb-86c3-8e513cdfee8f admin admin] PATH_INFO: `/v3/projects/5c215d2f45d8468a870b116e180233d0` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:19.720887 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b441eac-6bcb-49eb-86c3-8e513cdfee8f admin admin] RBAC: Authorizing `identity:delete_project(project_id=5c215d2f45d8468a870b116e180233d0)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:19.725763 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b441eac-6bcb-49eb-86c3-8e513cdfee8f admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:19.799110 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76dcfe03-348f-49ab-ba3d-f3f9444c5639 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:19.801411 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76dcfe03-348f-49ab-ba3d-f3f9444c5639 service cinder] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:19.821879 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1086/2168] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:35:19 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 296 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 21:35:19.860493 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76dcfe03-348f-49ab-ba3d-f3f9444c5639 service cinder] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:19.863771 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-76dcfe03-348f-49ab-ba3d-f3f9444c5639 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:19.866305 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-2b441eac-6bcb-49eb-86c3-8e513cdfee8f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5c215d2f45d8468a870b116e180233d0', 'request_id': 'req-2b441eac-6bcb-49eb-86c3-8e513cdfee8f'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:19.897074 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-2b441eac-6bcb-49eb-86c3-8e513cdfee8f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5c215d2f45d8468a870b116e180233d0', 'request_id': 'req-2b441eac-6bcb-49eb-86c3-8e513cdfee8f'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:19.905561 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-2b441eac-6bcb-49eb-86c3-8e513cdfee8f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5c215d2f45d8468a870b116e180233d0', 'request_id': 'req-2b441eac-6bcb-49eb-86c3-8e513cdfee8f'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:19.934962 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-2b441eac-6bcb-49eb-86c3-8e513cdfee8f admin admin] The token cache is being invalidate because project 5c215d2f45d8468a870b116e180233d0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:35:19.935622 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-2b441eac-6bcb-49eb-86c3-8e513cdfee8f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:19.936697 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-2b441eac-6bcb-49eb-86c3-8e513cdfee8f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:19.951560 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1083/2169] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:35:19 2022] DELETE /identity/v3/projects/5c215d2f45d8468a870b116e180233d0 => generated 0 bytes in 308 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 21:35:19.956808 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b441eac-6bcb-49eb-86c3-8e513cdfee8f admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:19.959449 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-2b441eac-6bcb-49eb-86c3-8e513cdfee8f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:20.015985 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76dcfe03-348f-49ab-ba3d-f3f9444c5639 service cinder] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:20.057504 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:35:20.159960 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b6227d4-7c1c-40e2-8d84-6cf252d0afa6 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a35b5b7dea7b421da056f063d2436ffb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:20.160946 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b6227d4-7c1c-40e2-8d84-6cf252d0afa6 service nova] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:20.161339 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b6227d4-7c1c-40e2-8d84-6cf252d0afa6 service nova] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:20.161733 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b6227d4-7c1c-40e2-8d84-6cf252d0afa6 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:20.164919 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-7b6227d4-7c1c-40e2-8d84-6cf252d0afa6 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:20.188849 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b441eac-6bcb-49eb-86c3-8e513cdfee8f admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:20.203629 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-350b3240-f745-4831-8ec9-ab3b95411415 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:20.204123 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-350b3240-f745-4831-8ec9-ab3b95411415 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:20.204356 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-350b3240-f745-4831-8ec9-ab3b95411415 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:20.204677 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-350b3240-f745-4831-8ec9-ab3b95411415 admin admin] PATH_INFO: `/v3/users/8f466fa1017840d58019b0d2d2ce738c` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:20.205955 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-350b3240-f745-4831-8ec9-ab3b95411415 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8f466fa1017840d58019b0d2d2ce738c)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:20.208620 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-350b3240-f745-4831-8ec9-ab3b95411415 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:20.320833 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-350b3240-f745-4831-8ec9-ab3b95411415 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8f466fa1017840d58019b0d2d2ce738c'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:20.378959 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-350b3240-f745-4831-8ec9-ab3b95411415 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8f466fa1017840d58019b0d2d2ce738c'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:20.379289 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b6227d4-7c1c-40e2-8d84-6cf252d0afa6 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:20.382834 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-350b3240-f745-4831-8ec9-ab3b95411415 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8f466fa1017840d58019b0d2d2ce738c'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:20.384440 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-350b3240-f745-4831-8ec9-ab3b95411415 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8f466fa1017840d58019b0d2d2ce738c'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:20.385168 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b6227d4-7c1c-40e2-8d84-6cf252d0afa6 service nova] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:20.405028 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-350b3240-f745-4831-8ec9-ab3b95411415 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8f466fa1017840d58019b0d2d2ce738c'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:20.409977 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=83,keystone:DELETE=16,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:35:20.419182 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1087/2170] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:35:19 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 560 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:35:20.449842 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1084/2171] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:35:19 2022] DELETE /identity/v3/users/8f466fa1017840d58019b0d2d2ce738c => generated 0 bytes in 494 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:35:20.716045 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b6227d4-7c1c-40e2-8d84-6cf252d0afa6 service nova] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:20.718003 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-7b6227d4-7c1c-40e2-8d84-6cf252d0afa6 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:20.760382 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b6227d4-7c1c-40e2-8d84-6cf252d0afa6 service nova] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:20.927022 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1af1418e-5b21-4d83-b3d4-3a4acbf95d2e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:20.928443 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1af1418e-5b21-4d83-b3d4-3a4acbf95d2e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:20.929073 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1af1418e-5b21-4d83-b3d4-3a4acbf95d2e admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:20.929590 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1af1418e-5b21-4d83-b3d4-3a4acbf95d2e admin admin] PATH_INFO: `/v3/users/e9e14e1e61c64e3aac538de88162d183` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:20.932736 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1af1418e-5b21-4d83-b3d4-3a4acbf95d2e admin admin] RBAC: Authorizing `identity:delete_user(user_id=e9e14e1e61c64e3aac538de88162d183)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:20.936007 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1af1418e-5b21-4d83-b3d4-3a4acbf95d2e admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:21.077205 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-1af1418e-5b21-4d83-b3d4-3a4acbf95d2e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e9e14e1e61c64e3aac538de88162d183'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:21.139435 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-1af1418e-5b21-4d83-b3d4-3a4acbf95d2e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e9e14e1e61c64e3aac538de88162d183'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:21.146953 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:35:21.161580 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-1af1418e-5b21-4d83-b3d4-3a4acbf95d2e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e9e14e1e61c64e3aac538de88162d183'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:21.162585 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-1af1418e-5b21-4d83-b3d4-3a4acbf95d2e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e9e14e1e61c64e3aac538de88162d183'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:21.170217 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-1af1418e-5b21-4d83-b3d4-3a4acbf95d2e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e9e14e1e61c64e3aac538de88162d183'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:21.205712 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1088/2172] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:35:20 2022] DELETE /identity/v3/users/e9e14e1e61c64e3aac538de88162d183 => generated 0 bytes in 490 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 21:35:21.212616 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca6c7510-80b3-4872-9553-d84bcad5be4c None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:21.213268 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca6c7510-80b3-4872-9553-d84bcad5be4c None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:21.213268 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca6c7510-80b3-4872-9553-d84bcad5be4c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:21.272124 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-ca6c7510-80b3-4872-9553-d84bcad5be4c None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:35:21.273614 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-ca6c7510-80b3-4872-9553-d84bcad5be4c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:21.417064 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1085/2173] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:35:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 206 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:35:21.504558 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1af1418e-5b21-4d83-b3d4-3a4acbf95d2e admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:21.507228 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-1af1418e-5b21-4d83-b3d4-3a4acbf95d2e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:21.524155 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca6c7510-80b3-4872-9553-d84bcad5be4c None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:21.527952 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-ca6c7510-80b3-4872-9553-d84bcad5be4c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:21.591253 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca6c7510-80b3-4872-9553-d84bcad5be4c None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:21.604519 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-896a74a6-cbec-4273-a596-cbfb7c5d9d71 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:21.605751 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-896a74a6-cbec-4273-a596-cbfb7c5d9d71 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:21.607171 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-896a74a6-cbec-4273-a596-cbfb7c5d9d71 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:21.607519 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-896a74a6-cbec-4273-a596-cbfb7c5d9d71 admin admin] PATH_INFO: `/v3/projects/0034b8df1b904a0a92c173efa4b32fb8` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:21.609341 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-896a74a6-cbec-4273-a596-cbfb7c5d9d71 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0034b8df1b904a0a92c173efa4b32fb8)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:21.613403 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-896a74a6-cbec-4273-a596-cbfb7c5d9d71 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:21.778448 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1af1418e-5b21-4d83-b3d4-3a4acbf95d2e admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:21.780596 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-896a74a6-cbec-4273-a596-cbfb7c5d9d71 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0034b8df1b904a0a92c173efa4b32fb8', 'request_id': 'req-896a74a6-cbec-4273-a596-cbfb7c5d9d71'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:21.811652 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-896a74a6-cbec-4273-a596-cbfb7c5d9d71 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0034b8df1b904a0a92c173efa4b32fb8', 'request_id': 'req-896a74a6-cbec-4273-a596-cbfb7c5d9d71'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:21.822640 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-896a74a6-cbec-4273-a596-cbfb7c5d9d71 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0034b8df1b904a0a92c173efa4b32fb8', 'request_id': 'req-896a74a6-cbec-4273-a596-cbfb7c5d9d71'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:21.853021 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-896a74a6-cbec-4273-a596-cbfb7c5d9d71 admin admin] The token cache is being invalidate because project 0034b8df1b904a0a92c173efa4b32fb8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:35:21.853550 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-896a74a6-cbec-4273-a596-cbfb7c5d9d71 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:21.860291 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-896a74a6-cbec-4273-a596-cbfb7c5d9d71 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:21.870252 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1086/2174] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:35:21 2022] DELETE /identity/v3/projects/0034b8df1b904a0a92c173efa4b32fb8 => generated 0 bytes in 347 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:35:21.879153 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-896a74a6-cbec-4273-a596-cbfb7c5d9d71 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:21.881149 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-896a74a6-cbec-4273-a596-cbfb7c5d9d71 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:21.946617 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fdd9082-51ec-42fd-b1e5-97a1920061b3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:21.949340 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fdd9082-51ec-42fd-b1e5-97a1920061b3 service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:21.949916 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fdd9082-51ec-42fd-b1e5-97a1920061b3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:21.950380 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fdd9082-51ec-42fd-b1e5-97a1920061b3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:21.959286 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-9fdd9082-51ec-42fd-b1e5-97a1920061b3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:22.108888 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-896a74a6-cbec-4273-a596-cbfb7c5d9d71 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:22.130575 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1abcf489-774e-43aa-8b09-afebdfe5aa78 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:22.131390 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1abcf489-774e-43aa-8b09-afebdfe5aa78 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:22.131782 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1abcf489-774e-43aa-8b09-afebdfe5aa78 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:22.132266 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1abcf489-774e-43aa-8b09-afebdfe5aa78 admin admin] PATH_INFO: `/v3/users/dd83be06650843e29927eba212e5140a` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:22.133893 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1abcf489-774e-43aa-8b09-afebdfe5aa78 admin admin] RBAC: Authorizing `identity:delete_user(user_id=dd83be06650843e29927eba212e5140a)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:22.136144 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1abcf489-774e-43aa-8b09-afebdfe5aa78 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:22.178896 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fdd9082-51ec-42fd-b1e5-97a1920061b3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:22.181343 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fdd9082-51ec-42fd-b1e5-97a1920061b3 service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:22.183396 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:INSERT=2,keystone:DELETE=9,keystone:SELECT=89 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:35:22.216581 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1089/2175] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:35:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 713 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 21:35:22.304474 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-1abcf489-774e-43aa-8b09-afebdfe5aa78 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'dd83be06650843e29927eba212e5140a'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:22.363172 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-1abcf489-774e-43aa-8b09-afebdfe5aa78 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'dd83be06650843e29927eba212e5140a'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:22.365829 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-1abcf489-774e-43aa-8b09-afebdfe5aa78 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'dd83be06650843e29927eba212e5140a'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:22.370315 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-1abcf489-774e-43aa-8b09-afebdfe5aa78 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'dd83be06650843e29927eba212e5140a'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:22.391903 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-1abcf489-774e-43aa-8b09-afebdfe5aa78 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'dd83be06650843e29927eba212e5140a'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:22.433997 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1087/2176] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:35:21 2022] DELETE /identity/v3/users/dd83be06650843e29927eba212e5140a => generated 0 bytes in 558 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:35:23.248680 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fdd9082-51ec-42fd-b1e5-97a1920061b3 service neutron] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:23.252181 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-9fdd9082-51ec-42fd-b1e5-97a1920061b3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:23.282696 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fdd9082-51ec-42fd-b1e5-97a1920061b3 service neutron] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:23.435196 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:DELETE=3,keystone:SELECT=97 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:35:23.498238 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-144c8cff-c482-4cbb-bdae-4cb270d2000b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:23.498906 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-144c8cff-c482-4cbb-bdae-4cb270d2000b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:23.499177 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-144c8cff-c482-4cbb-bdae-4cb270d2000b admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:23.499454 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-144c8cff-c482-4cbb-bdae-4cb270d2000b admin admin] PATH_INFO: `/v3/projects/cac54fe0108d448abb161656c9c50bdd` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:23.500703 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-144c8cff-c482-4cbb-bdae-4cb270d2000b admin admin] RBAC: Authorizing `identity:delete_project(project_id=cac54fe0108d448abb161656c9c50bdd)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:23.504512 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-144c8cff-c482-4cbb-bdae-4cb270d2000b admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:23.653593 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-144c8cff-c482-4cbb-bdae-4cb270d2000b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cac54fe0108d448abb161656c9c50bdd', 'request_id': 'req-144c8cff-c482-4cbb-bdae-4cb270d2000b'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:23.667664 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-144c8cff-c482-4cbb-bdae-4cb270d2000b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cac54fe0108d448abb161656c9c50bdd', 'request_id': 'req-144c8cff-c482-4cbb-bdae-4cb270d2000b'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:23.682603 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-144c8cff-c482-4cbb-bdae-4cb270d2000b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cac54fe0108d448abb161656c9c50bdd', 'request_id': 'req-144c8cff-c482-4cbb-bdae-4cb270d2000b'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:23.712667 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-144c8cff-c482-4cbb-bdae-4cb270d2000b admin admin] The token cache is being invalidate because project cac54fe0108d448abb161656c9c50bdd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81434) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:35:23.715420 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-144c8cff-c482-4cbb-bdae-4cb270d2000b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:23.717740 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-144c8cff-c482-4cbb-bdae-4cb270d2000b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:23.726216 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1090/2177] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:35:23 2022] DELETE /identity/v3/projects/cac54fe0108d448abb161656c9c50bdd => generated 0 bytes in 478 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:35:23.735086 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e169fd1-dc5f-4794-9a54-2a0b811a140c None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:23.736345 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e169fd1-dc5f-4794-9a54-2a0b811a140c None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:23.737932 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e169fd1-dc5f-4794-9a54-2a0b811a140c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:23.827123 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-9e169fd1-dc5f-4794-9a54-2a0b811a140c None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:35:23.828805 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-9e169fd1-dc5f-4794-9a54-2a0b811a140c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:23.839334 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-144c8cff-c482-4cbb-bdae-4cb270d2000b admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:23.842082 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-144c8cff-c482-4cbb-bdae-4cb270d2000b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:24.083370 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1088/2178] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:35:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 351 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:35:24.089883 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-144c8cff-c482-4cbb-bdae-4cb270d2000b admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:24.092641 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00eecef2-a999-4b55-8f0f-6ed8f3fb0c0a None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:24.093135 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00eecef2-a999-4b55-8f0f-6ed8f3fb0c0a None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:24.093619 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00eecef2-a999-4b55-8f0f-6ed8f3fb0c0a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:24.136738 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ab898f0-da30-4319-b113-170d58cff17f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:24.137425 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ab898f0-da30-4319-b113-170d58cff17f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:24.138041 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ab898f0-da30-4319-b113-170d58cff17f admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:24.140393 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ab898f0-da30-4319-b113-170d58cff17f admin admin] PATH_INFO: `/v3/projects/6a31ec5d4f7f4af2abb2cfd0848d26ff` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:24.148869 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ab898f0-da30-4319-b113-170d58cff17f admin admin] RBAC: Authorizing `identity:delete_project(project_id=6a31ec5d4f7f4af2abb2cfd0848d26ff)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:24.153698 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ab898f0-da30-4319-b113-170d58cff17f admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:24.173494 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-00eecef2-a999-4b55-8f0f-6ed8f3fb0c0a None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:35:24.176895 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-00eecef2-a999-4b55-8f0f-6ed8f3fb0c0a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:24.210947 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1089/2179] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:35:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:35:24.217027 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00eecef2-a999-4b55-8f0f-6ed8f3fb0c0a None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:24.245661 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00eecef2-a999-4b55-8f0f-6ed8f3fb0c0a None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:24.265497 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61bbecf4-d114-4c21-bc5e-299e39e24e82 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:24.266626 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61bbecf4-d114-4c21-bc5e-299e39e24e82 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:24.267115 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61bbecf4-d114-4c21-bc5e-299e39e24e82 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:24.267589 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61bbecf4-d114-4c21-bc5e-299e39e24e82 admin admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:24.269478 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61bbecf4-d114-4c21-bc5e-299e39e24e82 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:35:24.270031 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61bbecf4-d114-4c21-bc5e-299e39e24e82 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:24.272942 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61bbecf4-d114-4c21-bc5e-299e39e24e82 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:24.309849 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-4ab898f0-da30-4319-b113-170d58cff17f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6a31ec5d4f7f4af2abb2cfd0848d26ff', 'request_id': 'req-4ab898f0-da30-4319-b113-170d58cff17f'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:24.326902 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1090/2180] 10.222.0.90 () {58 vars in 1153 bytes} [Mon Aug 29 21:35:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 111 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:35:24.334020 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61bbecf4-d114-4c21-bc5e-299e39e24e82 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:24.340738 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-4ab898f0-da30-4319-b113-170d58cff17f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6a31ec5d4f7f4af2abb2cfd0848d26ff', 'request_id': 'req-4ab898f0-da30-4319-b113-170d58cff17f'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:24.358281 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61bbecf4-d114-4c21-bc5e-299e39e24e82 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:24.358665 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-4ab898f0-da30-4319-b113-170d58cff17f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6a31ec5d4f7f4af2abb2cfd0848d26ff', 'request_id': 'req-4ab898f0-da30-4319-b113-170d58cff17f'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:24.398525 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-4ab898f0-da30-4319-b113-170d58cff17f admin admin] The token cache is being invalidate because project 6a31ec5d4f7f4af2abb2cfd0848d26ff was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81434) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:35:24.399236 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-4ab898f0-da30-4319-b113-170d58cff17f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:24.409868 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-4ab898f0-da30-4319-b113-170d58cff17f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:24.425875 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1091/2181] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:35:23 2022] DELETE /identity/v3/projects/6a31ec5d4f7f4af2abb2cfd0848d26ff => generated 0 bytes in 574 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:35:24.425875 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22cc7a65-d9d4-480a-a58c-59b5b865f76f None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:24.425875 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22cc7a65-d9d4-480a-a58c-59b5b865f76f None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:24.425875 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22cc7a65-d9d4-480a-a58c-59b5b865f76f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:24.459365 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:35:24.515265 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-22cc7a65-d9d4-480a-a58c-59b5b865f76f None None] MFA Rules not processed for user `b6a3df8d796e4ae6ac358f103903d427`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:35:24.526485 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-22cc7a65-d9d4-480a-a58c-59b5b865f76f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:24.663233 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-533a4c56-da1e-4261-9cca-63aa01768eae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:24.664298 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-533a4c56-da1e-4261-9cca-63aa01768eae admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:24.664778 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-533a4c56-da1e-4261-9cca-63aa01768eae admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:24.665248 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-533a4c56-da1e-4261-9cca-63aa01768eae admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:24.672874 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-533a4c56-da1e-4261-9cca-63aa01768eae admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:24.677501 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-533a4c56-da1e-4261-9cca-63aa01768eae admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:24.751993 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:DELETE=9,keystone:SELECT=89,keystone:INSERT=2 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:35:24.780139 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1091/2182] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:35:24 2022] POST /identity/v3/projects => generated 367 bytes in 447 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:35:24.784858 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-533a4c56-da1e-4261-9cca-63aa01768eae admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:24.787740 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-533a4c56-da1e-4261-9cca-63aa01768eae admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:24.832662 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1092/2183] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:35:24 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 412 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:35:24.962948 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-533a4c56-da1e-4261-9cca-63aa01768eae admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:24.969368 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55bb53ed-f2d9-468b-aea5-ab6b5c867241 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:24.970117 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55bb53ed-f2d9-468b-aea5-ab6b5c867241 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:24.970400 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55bb53ed-f2d9-468b-aea5-ab6b5c867241 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:24.970683 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55bb53ed-f2d9-468b-aea5-ab6b5c867241 admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:24.971448 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55bb53ed-f2d9-468b-aea5-ab6b5c867241 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:24.974792 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55bb53ed-f2d9-468b-aea5-ab6b5c867241 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:25.020134 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1092/2184] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:35:24 2022] POST /identity/v3/users => generated 284 bytes in 233 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 21:35:25.022195 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22cc7a65-d9d4-480a-a58c-59b5b865f76f None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:25.041143 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22cc7a65-d9d4-480a-a58c-59b5b865f76f None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:25.046929 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1d5ee8d-6e1b-4ff6-9cd3-d849fcb68c45 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:25.047697 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1d5ee8d-6e1b-4ff6-9cd3-d849fcb68c45 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:25.048052 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1d5ee8d-6e1b-4ff6-9cd3-d849fcb68c45 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:25.048536 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1d5ee8d-6e1b-4ff6-9cd3-d849fcb68c45 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:25.049300 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1d5ee8d-6e1b-4ff6-9cd3-d849fcb68c45 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:25.051374 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1d5ee8d-6e1b-4ff6-9cd3-d849fcb68c45 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:25.065847 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1093/2185] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:35:25 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:35:25.072581 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55bb53ed-f2d9-468b-aea5-ab6b5c867241 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:25.092654 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55bb53ed-f2d9-468b-aea5-ab6b5c867241 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:25.101748 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5563b307-ca39-4a0c-98d4-6bb10661e891 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:25.102866 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5563b307-ca39-4a0c-98d4-6bb10661e891 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:25.103441 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5563b307-ca39-4a0c-98d4-6bb10661e891 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:25.103976 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5563b307-ca39-4a0c-98d4-6bb10661e891 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:25.105028 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5563b307-ca39-4a0c-98d4-6bb10661e891 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:25.107974 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5563b307-ca39-4a0c-98d4-6bb10661e891 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:25.122483 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1093/2186] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:35:25 2022] GET /identity/v3/roles => generated 1420 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:35:25.126656 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1d5ee8d-6e1b-4ff6-9cd3-d849fcb68c45 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:25.149867 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1d5ee8d-6e1b-4ff6-9cd3-d849fcb68c45 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:25.159308 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6a52c40-8910-4fa1-b0b0-3cd9a3b535f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:25.161175 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6a52c40-8910-4fa1-b0b0-3cd9a3b535f4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:25.161612 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6a52c40-8910-4fa1-b0b0-3cd9a3b535f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:25.162000 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6a52c40-8910-4fa1-b0b0-3cd9a3b535f4 admin admin] PATH_INFO: `/v3/projects/33a79f7ecf224e03ae0e5bf5e12a7638/users/1799b582f9a5462fbcd671b9415da8d3/roles/02e242d5520740d4b77a5f4008c7eb57` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:25.234075 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6a52c40-8910-4fa1-b0b0-3cd9a3b535f4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=33a79f7ecf224e03ae0e5bf5e12a7638, user_id=1799b582f9a5462fbcd671b9415da8d3, role_id=02e242d5520740d4b77a5f4008c7eb57)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:25.237945 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6a52c40-8910-4fa1-b0b0-3cd9a3b535f4 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:25.238306 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5563b307-ca39-4a0c-98d4-6bb10661e891 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:25.240299 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-5563b307-ca39-4a0c-98d4-6bb10661e891 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:25.263328 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1094/2187] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:35:25 2022] PUT /identity/v3/projects/33a79f7ecf224e03ae0e5bf5e12a7638/users/1799b582f9a5462fbcd671b9415da8d3/roles/02e242d5520740d4b77a5f4008c7eb57 => generated 0 bytes in 137 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 21:35:25.276982 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-628bdc14-74e7-4c8c-9bd4-37541b6e8173 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:25.277351 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5563b307-ca39-4a0c-98d4-6bb10661e891 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:25.277982 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-628bdc14-74e7-4c8c-9bd4-37541b6e8173 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:25.278911 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-628bdc14-74e7-4c8c-9bd4-37541b6e8173 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:25.318739 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-628bdc14-74e7-4c8c-9bd4-37541b6e8173 None None] MFA Rules not processed for user `1799b582f9a5462fbcd671b9415da8d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:35:25.320187 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-628bdc14-74e7-4c8c-9bd4-37541b6e8173 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:25.392819 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e31f24c8-1c9a-4e80-8c26-2bef19302e8a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:25.394348 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e31f24c8-1c9a-4e80-8c26-2bef19302e8a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:25.394804 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e31f24c8-1c9a-4e80-8c26-2bef19302e8a admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:25.395241 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e31f24c8-1c9a-4e80-8c26-2bef19302e8a admin admin] PATH_INFO: `/v3/users/b5e69fef981244908ae6f53516e20bdf` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:25.397094 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e31f24c8-1c9a-4e80-8c26-2bef19302e8a admin admin] RBAC: Authorizing `identity:delete_user(user_id=b5e69fef981244908ae6f53516e20bdf)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:25.399825 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e31f24c8-1c9a-4e80-8c26-2bef19302e8a admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:25.433464 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1095/2188] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:35:25 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 21:35:25.470758 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-628bdc14-74e7-4c8c-9bd4-37541b6e8173 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:25.475287 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-628bdc14-74e7-4c8c-9bd4-37541b6e8173 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:25.482647 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-e31f24c8-1c9a-4e80-8c26-2bef19302e8a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b5e69fef981244908ae6f53516e20bdf'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:25.515154 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-e31f24c8-1c9a-4e80-8c26-2bef19302e8a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b5e69fef981244908ae6f53516e20bdf'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:25.516329 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-e31f24c8-1c9a-4e80-8c26-2bef19302e8a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b5e69fef981244908ae6f53516e20bdf'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:25.517290 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-e31f24c8-1c9a-4e80-8c26-2bef19302e8a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b5e69fef981244908ae6f53516e20bdf'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:25.531516 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-e31f24c8-1c9a-4e80-8c26-2bef19302e8a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b5e69fef981244908ae6f53516e20bdf'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:25.560325 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=86,keystone:INSERT=4,keystone:DELETE=10 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:35:25.574594 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1094/2189] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:35:25 2022] DELETE /identity/v3/users/b5e69fef981244908ae6f53516e20bdf => generated 0 bytes in 337 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 21:35:25.586183 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f42b657-74bc-4544-a19d-abdd439e1bcf None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:25.587378 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f42b657-74bc-4544-a19d-abdd439e1bcf None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:25.588013 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f42b657-74bc-4544-a19d-abdd439e1bcf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:25.622042 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-7f42b657-74bc-4544-a19d-abdd439e1bcf None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:35:25.624873 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-7f42b657-74bc-4544-a19d-abdd439e1bcf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:25.637735 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:35:25.660517 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-628bdc14-74e7-4c8c-9bd4-37541b6e8173 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:25.727316 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1095/2190] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:35:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 147 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:35:25.740463 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f42b657-74bc-4544-a19d-abdd439e1bcf None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:25.741638 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-7f42b657-74bc-4544-a19d-abdd439e1bcf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:25.755193 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-110dfb53-5295-4cec-81f7-0772e0a7b06b service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c63a4eaf48ad43529f0aa328f2ebfd29', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:25.756113 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-110dfb53-5295-4cec-81f7-0772e0a7b06b service glance] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:25.756394 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-110dfb53-5295-4cec-81f7-0772e0a7b06b service glance] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:25.756687 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-110dfb53-5295-4cec-81f7-0772e0a7b06b service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:25.759832 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-110dfb53-5295-4cec-81f7-0772e0a7b06b service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:25.864100 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f42b657-74bc-4544-a19d-abdd439e1bcf None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:25.884230 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3b72b24-d8e7-4812-99b2-9a0470957998 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:25.885171 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3b72b24-d8e7-4812-99b2-9a0470957998 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:25.885627 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3b72b24-d8e7-4812-99b2-9a0470957998 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:25.886130 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3b72b24-d8e7-4812-99b2-9a0470957998 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:25.917632 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3b72b24-d8e7-4812-99b2-9a0470957998 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:25.918195 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-110dfb53-5295-4cec-81f7-0772e0a7b06b service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:25.921078 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-110dfb53-5295-4cec-81f7-0772e0a7b06b service glance] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:25.937385 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3b72b24-d8e7-4812-99b2-9a0470957998 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:25.959834 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1096/2191] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:35:25 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 495 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:35:25.985380 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1096/2192] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:35:25 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 241 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 21:35:27.060501 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-110dfb53-5295-4cec-81f7-0772e0a7b06b service glance] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:27.068933 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-110dfb53-5295-4cec-81f7-0772e0a7b06b service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:27.109280 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-110dfb53-5295-4cec-81f7-0772e0a7b06b service glance] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:27.123466 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ce1c44e-631e-44cf-9a2a-dcfa4c4c0987 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:27.124772 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ce1c44e-631e-44cf-9a2a-dcfa4c4c0987 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:27.125351 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ce1c44e-631e-44cf-9a2a-dcfa4c4c0987 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:27.125945 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ce1c44e-631e-44cf-9a2a-dcfa4c4c0987 admin admin] PATH_INFO: `/v3/projects/44beb31da9fb4270ac4cfcd7fd987fd4` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:27.129743 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ce1c44e-631e-44cf-9a2a-dcfa4c4c0987 admin admin] RBAC: Authorizing `identity:delete_project(project_id=44beb31da9fb4270ac4cfcd7fd987fd4)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:27.137215 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ce1c44e-631e-44cf-9a2a-dcfa4c4c0987 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:27.237995 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-9ce1c44e-631e-44cf-9a2a-dcfa4c4c0987 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '44beb31da9fb4270ac4cfcd7fd987fd4', 'request_id': 'req-9ce1c44e-631e-44cf-9a2a-dcfa4c4c0987'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:27.262104 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-9ce1c44e-631e-44cf-9a2a-dcfa4c4c0987 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '44beb31da9fb4270ac4cfcd7fd987fd4', 'request_id': 'req-9ce1c44e-631e-44cf-9a2a-dcfa4c4c0987'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:27.278090 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-9ce1c44e-631e-44cf-9a2a-dcfa4c4c0987 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '44beb31da9fb4270ac4cfcd7fd987fd4', 'request_id': 'req-9ce1c44e-631e-44cf-9a2a-dcfa4c4c0987'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:27.307690 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-9ce1c44e-631e-44cf-9a2a-dcfa4c4c0987 admin admin] The token cache is being invalidate because project 44beb31da9fb4270ac4cfcd7fd987fd4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81434) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:35:27.308071 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-9ce1c44e-631e-44cf-9a2a-dcfa4c4c0987 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:27.309509 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-9ce1c44e-631e-44cf-9a2a-dcfa4c4c0987 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:27.313029 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1097/2193] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:35:27 2022] DELETE /identity/v3/projects/44beb31da9fb4270ac4cfcd7fd987fd4 => generated 0 bytes in 254 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:35:27.320743 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3b72b24-d8e7-4812-99b2-9a0470957998 service neutron] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:27.336837 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3b72b24-d8e7-4812-99b2-9a0470957998 service neutron] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:27.453921 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a255698-e14e-4ded-8a7e-c02bf4c7488d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:27.455006 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a255698-e14e-4ded-8a7e-c02bf4c7488d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:27.455508 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a255698-e14e-4ded-8a7e-c02bf4c7488d admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:27.457493 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a255698-e14e-4ded-8a7e-c02bf4c7488d admin admin] PATH_INFO: `/v3/users/c7aff8ab512d470ebedea4aaa327e0b5` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:27.459960 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a255698-e14e-4ded-8a7e-c02bf4c7488d admin admin] RBAC: Authorizing `identity:delete_user(user_id=c7aff8ab512d470ebedea4aaa327e0b5)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:27.464097 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a255698-e14e-4ded-8a7e-c02bf4c7488d admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:27.561610 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=94,keystone:DELETE=6 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:35:27.567162 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-8a255698-e14e-4ded-8a7e-c02bf4c7488d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c7aff8ab512d470ebedea4aaa327e0b5'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:27.603278 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-8a255698-e14e-4ded-8a7e-c02bf4c7488d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c7aff8ab512d470ebedea4aaa327e0b5'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:27.604563 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-8a255698-e14e-4ded-8a7e-c02bf4c7488d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c7aff8ab512d470ebedea4aaa327e0b5'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:27.605675 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-8a255698-e14e-4ded-8a7e-c02bf4c7488d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c7aff8ab512d470ebedea4aaa327e0b5'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:27.627309 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-8a255698-e14e-4ded-8a7e-c02bf4c7488d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c7aff8ab512d470ebedea4aaa327e0b5'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:27.667019 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1097/2194] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:35:27 2022] DELETE /identity/v3/users/c7aff8ab512d470ebedea4aaa327e0b5 => generated 0 bytes in 347 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 21:35:28.608579 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ce1c44e-631e-44cf-9a2a-dcfa4c4c0987 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:28.610214 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-9ce1c44e-631e-44cf-9a2a-dcfa4c4c0987 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:28.629630 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ce1c44e-631e-44cf-9a2a-dcfa4c4c0987 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:28.705305 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:35:28.750321 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b483faaa-8588-4840-aa70-6ecc0ae72141 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:28.750971 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b483faaa-8588-4840-aa70-6ecc0ae72141 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:28.751254 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b483faaa-8588-4840-aa70-6ecc0ae72141 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:28.751536 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b483faaa-8588-4840-aa70-6ecc0ae72141 admin admin] PATH_INFO: `/v3/projects/c61d110514dc41648a9c202238496fa7` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:28.753305 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b483faaa-8588-4840-aa70-6ecc0ae72141 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c61d110514dc41648a9c202238496fa7)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:28.755250 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b483faaa-8588-4840-aa70-6ecc0ae72141 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:28.819249 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-b483faaa-8588-4840-aa70-6ecc0ae72141 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c61d110514dc41648a9c202238496fa7', 'request_id': 'req-b483faaa-8588-4840-aa70-6ecc0ae72141'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:28.829294 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-b483faaa-8588-4840-aa70-6ecc0ae72141 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c61d110514dc41648a9c202238496fa7', 'request_id': 'req-b483faaa-8588-4840-aa70-6ecc0ae72141'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:28.842148 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-b483faaa-8588-4840-aa70-6ecc0ae72141 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c61d110514dc41648a9c202238496fa7', 'request_id': 'req-b483faaa-8588-4840-aa70-6ecc0ae72141'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:28.859898 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-b483faaa-8588-4840-aa70-6ecc0ae72141 admin admin] The token cache is being invalidate because project c61d110514dc41648a9c202238496fa7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81434) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:35:28.860982 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-b483faaa-8588-4840-aa70-6ecc0ae72141 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:28.861300 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-b483faaa-8588-4840-aa70-6ecc0ae72141 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:28.867369 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1098/2195] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:35:28 2022] DELETE /identity/v3/projects/c61d110514dc41648a9c202238496fa7 => generated 0 bytes in 256 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 21:35:28.869400 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a255698-e14e-4ded-8a7e-c02bf4c7488d admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:28.871682 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-8a255698-e14e-4ded-8a7e-c02bf4c7488d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:28.886253 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a255698-e14e-4ded-8a7e-c02bf4c7488d admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:28.993267 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38c9cc5f-0718-4ef4-8327-ea2d3266b96e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:28.995687 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38c9cc5f-0718-4ef4-8327-ea2d3266b96e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:28.996152 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38c9cc5f-0718-4ef4-8327-ea2d3266b96e admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:28.996903 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38c9cc5f-0718-4ef4-8327-ea2d3266b96e admin admin] PATH_INFO: `/v3/users/f2f5aa779f1d46fcb2ac046a214992dc` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:28.999677 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38c9cc5f-0718-4ef4-8327-ea2d3266b96e admin admin] RBAC: Authorizing `identity:delete_user(user_id=f2f5aa779f1d46fcb2ac046a214992dc)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:29.004574 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38c9cc5f-0718-4ef4-8327-ea2d3266b96e admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:29.087534 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-38c9cc5f-0718-4ef4-8327-ea2d3266b96e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f2f5aa779f1d46fcb2ac046a214992dc'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:29.116290 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-38c9cc5f-0718-4ef4-8327-ea2d3266b96e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f2f5aa779f1d46fcb2ac046a214992dc'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:29.120844 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-38c9cc5f-0718-4ef4-8327-ea2d3266b96e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f2f5aa779f1d46fcb2ac046a214992dc'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:29.122991 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-38c9cc5f-0718-4ef4-8327-ea2d3266b96e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f2f5aa779f1d46fcb2ac046a214992dc'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:29.139832 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-38c9cc5f-0718-4ef4-8327-ea2d3266b96e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f2f5aa779f1d46fcb2ac046a214992dc'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:29.175665 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1098/2196] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:35:28 2022] DELETE /identity/v3/users/f2f5aa779f1d46fcb2ac046a214992dc => generated 0 bytes in 307 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:35:29.624503 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b483faaa-8588-4840-aa70-6ecc0ae72141 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:29.626716 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-b483faaa-8588-4840-aa70-6ecc0ae72141 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:29.659599 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b483faaa-8588-4840-aa70-6ecc0ae72141 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:29.803736 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7691e262-ec8e-4e85-9894-db11cebe862b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:29.806707 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7691e262-ec8e-4e85-9894-db11cebe862b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:29.807163 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7691e262-ec8e-4e85-9894-db11cebe862b admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:29.807614 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7691e262-ec8e-4e85-9894-db11cebe862b admin admin] PATH_INFO: `/v3/projects/84397e7a5d0745be8ee33ad50558cafa` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:29.810800 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7691e262-ec8e-4e85-9894-db11cebe862b admin admin] RBAC: Authorizing `identity:delete_project(project_id=84397e7a5d0745be8ee33ad50558cafa)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:29.814297 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7691e262-ec8e-4e85-9894-db11cebe862b admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:29.899262 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-7691e262-ec8e-4e85-9894-db11cebe862b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '84397e7a5d0745be8ee33ad50558cafa', 'request_id': 'req-7691e262-ec8e-4e85-9894-db11cebe862b'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:29.914591 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-7691e262-ec8e-4e85-9894-db11cebe862b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '84397e7a5d0745be8ee33ad50558cafa', 'request_id': 'req-7691e262-ec8e-4e85-9894-db11cebe862b'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:29.932443 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-7691e262-ec8e-4e85-9894-db11cebe862b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '84397e7a5d0745be8ee33ad50558cafa', 'request_id': 'req-7691e262-ec8e-4e85-9894-db11cebe862b'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:29.939103 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:35:29.953137 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-7691e262-ec8e-4e85-9894-db11cebe862b admin admin] The token cache is being invalidate because project 84397e7a5d0745be8ee33ad50558cafa was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81434) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:35:29.953837 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-7691e262-ec8e-4e85-9894-db11cebe862b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:29.954580 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-7691e262-ec8e-4e85-9894-db11cebe862b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:29.960211 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1099/2197] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:35:29 2022] DELETE /identity/v3/projects/84397e7a5d0745be8ee33ad50558cafa => generated 0 bytes in 337 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:35:29.969760 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b6b820-5d72-4537-a203-4eb3ba52da37 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:29.970609 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b6b820-5d72-4537-a203-4eb3ba52da37 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:29.970806 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b6b820-5d72-4537-a203-4eb3ba52da37 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:29.996133 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-41b6b820-5d72-4537-a203-4eb3ba52da37 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:35:29.999075 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-41b6b820-5d72-4537-a203-4eb3ba52da37 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:30.051366 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:DELETE=14,keystone:SELECT=84,keystone:INSERT=2 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:35:30.097976 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1099/2198] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:35:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:35:30.103689 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e985daa3-8b69-412b-ac45-db80fa1c56b4 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:30.105595 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e985daa3-8b69-412b-ac45-db80fa1c56b4 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:30.106235 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e985daa3-8b69-412b-ac45-db80fa1c56b4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:30.141121 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-e985daa3-8b69-412b-ac45-db80fa1c56b4 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:35:30.142395 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-e985daa3-8b69-412b-ac45-db80fa1c56b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:30.159766 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1100/2199] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:35:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 58 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:35:30.165674 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41b6b820-5d72-4537-a203-4eb3ba52da37 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:30.181475 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41b6b820-5d72-4537-a203-4eb3ba52da37 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:30.187814 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-820b785c-2ecd-4079-9672-879deb2498e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:30.187814 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-820b785c-2ecd-4079-9672-879deb2498e5 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:30.187814 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-820b785c-2ecd-4079-9672-879deb2498e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:30.190748 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-820b785c-2ecd-4079-9672-879deb2498e5 admin admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:30.190748 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-820b785c-2ecd-4079-9672-879deb2498e5 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:35:30.190748 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-820b785c-2ecd-4079-9672-879deb2498e5 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:30.191209 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-820b785c-2ecd-4079-9672-879deb2498e5 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:30.208407 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1100/2200] 10.222.0.90 () {58 vars in 1153 bytes} [Mon Aug 29 21:35:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:35:30.213584 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e985daa3-8b69-412b-ac45-db80fa1c56b4 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:30.232552 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e985daa3-8b69-412b-ac45-db80fa1c56b4 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:30.242424 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dd85d9b-b22e-4214-b1a1-c08df102e5e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:30.243260 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd85d9b-b22e-4214-b1a1-c08df102e5e0 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:30.243715 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd85d9b-b22e-4214-b1a1-c08df102e5e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:30.244141 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd85d9b-b22e-4214-b1a1-c08df102e5e0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:30.245073 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dd85d9b-b22e-4214-b1a1-c08df102e5e0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:30.247370 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dd85d9b-b22e-4214-b1a1-c08df102e5e0 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:30.289287 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1101/2201] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:35:30 2022] POST /identity/v3/projects => generated 361 bytes in 76 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:35:30.294712 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-820b785c-2ecd-4079-9672-879deb2498e5 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:30.311893 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-820b785c-2ecd-4079-9672-879deb2498e5 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:30.394483 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7234b4a5-9596-4bdd-91e6-31477a0c1a19 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:30.395416 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7234b4a5-9596-4bdd-91e6-31477a0c1a19 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:30.395845 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7234b4a5-9596-4bdd-91e6-31477a0c1a19 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:30.396172 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7234b4a5-9596-4bdd-91e6-31477a0c1a19 admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:30.397061 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7234b4a5-9596-4bdd-91e6-31477a0c1a19 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:30.399269 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7234b4a5-9596-4bdd-91e6-31477a0c1a19 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:30.437475 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1101/2202] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:35:30 2022] POST /identity/v3/users => generated 281 bytes in 144 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:35:30.443335 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dd85d9b-b22e-4214-b1a1-c08df102e5e0 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:30.463168 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dd85d9b-b22e-4214-b1a1-c08df102e5e0 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:30.470222 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cb1b0f8-a5e6-4fd1-9a79-f0648dc9c34e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:30.470849 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cb1b0f8-a5e6-4fd1-9a79-f0648dc9c34e admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:30.471639 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cb1b0f8-a5e6-4fd1-9a79-f0648dc9c34e admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:30.471912 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cb1b0f8-a5e6-4fd1-9a79-f0648dc9c34e admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:30.472684 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cb1b0f8-a5e6-4fd1-9a79-f0648dc9c34e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:30.474595 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cb1b0f8-a5e6-4fd1-9a79-f0648dc9c34e admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:30.494952 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1102/2203] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:35:30 2022] GET /identity/v3/roles => generated 1420 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:35:30.506165 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7234b4a5-9596-4bdd-91e6-31477a0c1a19 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:30.525748 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7234b4a5-9596-4bdd-91e6-31477a0c1a19 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:30.531249 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b06dd3ef-8228-40b6-b273-08eea3bd33b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:30.531985 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b06dd3ef-8228-40b6-b273-08eea3bd33b0 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:30.532429 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b06dd3ef-8228-40b6-b273-08eea3bd33b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:30.532888 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b06dd3ef-8228-40b6-b273-08eea3bd33b0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:30.535336 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b06dd3ef-8228-40b6-b273-08eea3bd33b0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:30.538649 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b06dd3ef-8228-40b6-b273-08eea3bd33b0 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:30.567507 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1102/2204] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:35:30 2022] GET /identity/v3/roles => generated 1420 bytes in 68 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:35:30.571919 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cb1b0f8-a5e6-4fd1-9a79-f0648dc9c34e admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:30.584909 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cb1b0f8-a5e6-4fd1-9a79-f0648dc9c34e admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:30.599476 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62dc2abf-328b-4371-a5e9-7b6064d3d154 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:30.600242 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62dc2abf-328b-4371-a5e9-7b6064d3d154 admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:30.600652 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62dc2abf-328b-4371-a5e9-7b6064d3d154 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:30.600841 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62dc2abf-328b-4371-a5e9-7b6064d3d154 admin admin] PATH_INFO: `/v3/projects/1faf244461314ecfab36d6f3d23c820c/users/6e285737d3d646389792cc57b912f475/roles/02e242d5520740d4b77a5f4008c7eb57` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:30.620596 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62dc2abf-328b-4371-a5e9-7b6064d3d154 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1faf244461314ecfab36d6f3d23c820c, user_id=6e285737d3d646389792cc57b912f475, role_id=02e242d5520740d4b77a5f4008c7eb57)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:30.623416 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62dc2abf-328b-4371-a5e9-7b6064d3d154 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:30.640939 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1103/2205] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:35:30 2022] PUT /identity/v3/projects/1faf244461314ecfab36d6f3d23c820c/users/6e285737d3d646389792cc57b912f475/roles/02e242d5520740d4b77a5f4008c7eb57 => generated 0 bytes in 70 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 21:35:30.652529 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c4cf75-0fcc-4667-877f-e2f4c820c81d None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:30.652748 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c4cf75-0fcc-4667-877f-e2f4c820c81d None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:30.652995 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c4cf75-0fcc-4667-877f-e2f4c820c81d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:30.701143 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-d6c4cf75-0fcc-4667-877f-e2f4c820c81d None None] MFA Rules not processed for user `6e285737d3d646389792cc57b912f475`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:35:30.703498 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-d6c4cf75-0fcc-4667-877f-e2f4c820c81d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:30.842981 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1103/2206] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:35:30 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 21:35:30.857724 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62dc2abf-328b-4371-a5e9-7b6064d3d154 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:30.860740 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-62dc2abf-328b-4371-a5e9-7b6064d3d154 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:30.959083 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62dc2abf-328b-4371-a5e9-7b6064d3d154 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:30.963828 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68dae5e7-6a82-49bb-ae18-c30d9683342c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dc226f7ddcb49ecacb51ce57fdfc11b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:30.964772 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68dae5e7-6a82-49bb-ae18-c30d9683342c service cinder] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:30.965052 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68dae5e7-6a82-49bb-ae18-c30d9683342c service cinder] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:30.965330 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68dae5e7-6a82-49bb-ae18-c30d9683342c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:30.980976 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68dae5e7-6a82-49bb-ae18-c30d9683342c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:30.982453 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68dae5e7-6a82-49bb-ae18-c30d9683342c service cinder] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:31.000852 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1104/2207] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:35:30 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 144 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:35:40.835150 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=81,keystone:INSERT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:35:40.988802 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=55,keystone:INSERT=2 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:35:48.523455 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e624dae3-d6d1-4ffa-a255-04e3ac783895 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:48.524684 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e624dae3-d6d1-4ffa-a255-04e3ac783895 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:48.524684 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e624dae3-d6d1-4ffa-a255-04e3ac783895 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:48.551667 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-e624dae3-d6d1-4ffa-a255-04e3ac783895 None None] MFA Rules not processed for user `0c0b0da87b124de1be00faec4bbdb404`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:35:48.553829 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-e624dae3-d6d1-4ffa-a255-04e3ac783895 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:48.668741 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1104/2208] 10.222.0.90 () {58 vars in 1009 bytes} [Mon Aug 29 21:35:48 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 147 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:35:48.740309 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68dae5e7-6a82-49bb-ae18-c30d9683342c service cinder] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:48.743585 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-68dae5e7-6a82-49bb-ae18-c30d9683342c service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:48.852047 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68dae5e7-6a82-49bb-ae18-c30d9683342c service cinder] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:48.858188 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e05afb48-494b-43d8-99dd-bccc0bad2075 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '139bbbde6d884b42b5440727926e2c58', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:48.858825 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e05afb48-494b-43d8-99dd-bccc0bad2075 service swift] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:48.859188 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e05afb48-494b-43d8-99dd-bccc0bad2075 service swift] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:48.859466 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e05afb48-494b-43d8-99dd-bccc0bad2075 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:48.869265 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e05afb48-494b-43d8-99dd-bccc0bad2075 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:48.871003 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e05afb48-494b-43d8-99dd-bccc0bad2075 service swift] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:48.883376 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1105/2209] 10.222.0.90 () {60 vars in 1384 bytes} [Mon Aug 29 21:35:48 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 144 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 29 21:35:49.707769 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e211e70-3a1a-4fc7-988f-0ab412e43ef5 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:49.708499 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e211e70-3a1a-4fc7-988f-0ab412e43ef5 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:49.708499 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e211e70-3a1a-4fc7-988f-0ab412e43ef5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:49.743991 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-1e211e70-3a1a-4fc7-988f-0ab412e43ef5 None None] MFA Rules not processed for user `0c0b0da87b124de1be00faec4bbdb404`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:35:49.745282 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-1e211e70-3a1a-4fc7-988f-0ab412e43ef5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:49.753828 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1105/2210] 10.222.0.90 () {58 vars in 967 bytes} [Mon Aug 29 21:35:49 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 47 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 21:35:49.779582 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e05afb48-494b-43d8-99dd-bccc0bad2075 service swift] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:49.797642 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e05afb48-494b-43d8-99dd-bccc0bad2075 service swift] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:49.805524 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63d1403c-d783-4079-98b1-f2b1dd13b8cb service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '139bbbde6d884b42b5440727926e2c58', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:49.806246 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63d1403c-d783-4079-98b1-f2b1dd13b8cb service swift] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:49.806533 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63d1403c-d783-4079-98b1-f2b1dd13b8cb service swift] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:49.806819 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63d1403c-d783-4079-98b1-f2b1dd13b8cb service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:49.820319 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63d1403c-d783-4079-98b1-f2b1dd13b8cb service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:49.821976 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63d1403c-d783-4079-98b1-f2b1dd13b8cb service swift] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:49.832588 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1106/2211] 10.222.0.90 () {60 vars in 1384 bytes} [Mon Aug 29 21:35:49 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 53 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 29 21:35:50.143867 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a65fae6-eac1-44b2-8ade-127ab2fb7e12 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:50.144175 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a65fae6-eac1-44b2-8ade-127ab2fb7e12 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:50.144455 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a65fae6-eac1-44b2-8ade-127ab2fb7e12 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:50.168015 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-2a65fae6-eac1-44b2-8ade-127ab2fb7e12 None None] MFA Rules not processed for user `0c0b0da87b124de1be00faec4bbdb404`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:35:50.169250 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-2a65fae6-eac1-44b2-8ade-127ab2fb7e12 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:50.178100 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1106/2212] 10.222.0.90 () {58 vars in 1009 bytes} [Mon Aug 29 21:35:50 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:35:50.195127 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63d1403c-d783-4079-98b1-f2b1dd13b8cb service swift] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:50.216557 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63d1403c-d783-4079-98b1-f2b1dd13b8cb service swift] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:50.223150 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d96313ab-1b84-4890-880f-d8b1d3dfce38 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '139bbbde6d884b42b5440727926e2c58', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:50.223815 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d96313ab-1b84-4890-880f-d8b1d3dfce38 service swift] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:50.224092 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d96313ab-1b84-4890-880f-d8b1d3dfce38 service swift] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:50.224374 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d96313ab-1b84-4890-880f-d8b1d3dfce38 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:50.236832 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d96313ab-1b84-4890-880f-d8b1d3dfce38 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:50.238406 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d96313ab-1b84-4890-880f-d8b1d3dfce38 service swift] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:50.248346 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1107/2213] 10.222.0.90 () {60 vars in 1384 bytes} [Mon Aug 29 21:35:50 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 54 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 29 21:35:50.665663 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4233f58-49e5-4bb0-9b60-149564dfa4ef None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:50.666274 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4233f58-49e5-4bb0-9b60-149564dfa4ef None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:50.666274 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4233f58-49e5-4bb0-9b60-149564dfa4ef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:50.686386 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-f4233f58-49e5-4bb0-9b60-149564dfa4ef None None] MFA Rules not processed for user `0c0b0da87b124de1be00faec4bbdb404`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:35:50.687662 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-f4233f58-49e5-4bb0-9b60-149564dfa4ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:50.697265 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1107/2214] 10.222.0.90 () {58 vars in 967 bytes} [Mon Aug 29 21:35:50 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:35:50.712570 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d96313ab-1b84-4890-880f-d8b1d3dfce38 service swift] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:50.724582 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d96313ab-1b84-4890-880f-d8b1d3dfce38 service swift] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:50.735663 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-393cc796-be15-4e6d-a1cd-412049e11bf7 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '139bbbde6d884b42b5440727926e2c58', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:50.736323 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-393cc796-be15-4e6d-a1cd-412049e11bf7 service swift] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:50.736633 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-393cc796-be15-4e6d-a1cd-412049e11bf7 service swift] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:50.737015 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-393cc796-be15-4e6d-a1cd-412049e11bf7 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:50.746489 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-393cc796-be15-4e6d-a1cd-412049e11bf7 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:50.748046 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-393cc796-be15-4e6d-a1cd-412049e11bf7 service swift] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:50.757001 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1108/2215] 10.222.0.90 () {60 vars in 1384 bytes} [Mon Aug 29 21:35:50 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 29 21:35:54.174421 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4233f58-49e5-4bb0-9b60-149564dfa4ef None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:54.176438 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-f4233f58-49e5-4bb0-9b60-149564dfa4ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:54.278373 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4233f58-49e5-4bb0-9b60-149564dfa4ef None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:54.284216 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb2c39c3-fd3f-4dca-a2db-34e54bbfd1b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:54.285710 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb2c39c3-fd3f-4dca-a2db-34e54bbfd1b0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:54.285710 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb2c39c3-fd3f-4dca-a2db-34e54bbfd1b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:54.285710 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb2c39c3-fd3f-4dca-a2db-34e54bbfd1b0 admin admin] PATH_INFO: `/v3/users/29a2a13867e7485590d34bb87b81d6f6` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:54.286521 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb2c39c3-fd3f-4dca-a2db-34e54bbfd1b0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=29a2a13867e7485590d34bb87b81d6f6)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:54.288468 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb2c39c3-fd3f-4dca-a2db-34e54bbfd1b0 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:54.343405 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-eb2c39c3-fd3f-4dca-a2db-34e54bbfd1b0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '29a2a13867e7485590d34bb87b81d6f6'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:54.357832 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-eb2c39c3-fd3f-4dca-a2db-34e54bbfd1b0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '29a2a13867e7485590d34bb87b81d6f6'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:54.358567 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-eb2c39c3-fd3f-4dca-a2db-34e54bbfd1b0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '29a2a13867e7485590d34bb87b81d6f6'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:54.359192 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-eb2c39c3-fd3f-4dca-a2db-34e54bbfd1b0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '29a2a13867e7485590d34bb87b81d6f6'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:54.367583 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-eb2c39c3-fd3f-4dca-a2db-34e54bbfd1b0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '29a2a13867e7485590d34bb87b81d6f6'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:54.385820 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1108/2216] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:35:54 2022] DELETE /identity/v3/users/29a2a13867e7485590d34bb87b81d6f6 => generated 0 bytes in 212 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:35:54.391631 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0df6efff-6d55-4c4f-90e8-ccceb780fa5a None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:54.392670 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0df6efff-6d55-4c4f-90e8-ccceb780fa5a None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:54.392991 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0df6efff-6d55-4c4f-90e8-ccceb780fa5a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:54.415544 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-0df6efff-6d55-4c4f-90e8-ccceb780fa5a None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:35:54.416832 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-0df6efff-6d55-4c4f-90e8-ccceb780fa5a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:54.502278 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1109/2217] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:35:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:35:54.513426 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb2c39c3-fd3f-4dca-a2db-34e54bbfd1b0 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:54.515013 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-eb2c39c3-fd3f-4dca-a2db-34e54bbfd1b0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:54.525641 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:35:54.588785 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb2c39c3-fd3f-4dca-a2db-34e54bbfd1b0 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:54.594431 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69c813a6-f07f-443f-a84b-39fd20a85c24 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:54.595867 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69c813a6-f07f-443f-a84b-39fd20a85c24 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:54.596250 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69c813a6-f07f-443f-a84b-39fd20a85c24 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:54.596528 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69c813a6-f07f-443f-a84b-39fd20a85c24 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:54.607491 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69c813a6-f07f-443f-a84b-39fd20a85c24 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:54.608933 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69c813a6-f07f-443f-a84b-39fd20a85c24 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:54.619058 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1109/2218] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:35:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:35:54.809952 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0df6efff-6d55-4c4f-90e8-ccceb780fa5a None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:54.811543 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-0df6efff-6d55-4c4f-90e8-ccceb780fa5a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:54.827096 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0df6efff-6d55-4c4f-90e8-ccceb780fa5a None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:54.849966 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-534fe5f2-c4f7-4507-b6b1-e598d538df10 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:54.852482 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-534fe5f2-c4f7-4507-b6b1-e598d538df10 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:54.852482 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-534fe5f2-c4f7-4507-b6b1-e598d538df10 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:54.853189 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-534fe5f2-c4f7-4507-b6b1-e598d538df10 admin admin] PATH_INFO: `/v3/projects/2dd38d98e8f042b69c60978ea8e7742b` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:54.855432 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-534fe5f2-c4f7-4507-b6b1-e598d538df10 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2dd38d98e8f042b69c60978ea8e7742b)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:54.859068 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-534fe5f2-c4f7-4507-b6b1-e598d538df10 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:54.931982 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:35:54.938388 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-534fe5f2-c4f7-4507-b6b1-e598d538df10 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2dd38d98e8f042b69c60978ea8e7742b', 'request_id': 'req-534fe5f2-c4f7-4507-b6b1-e598d538df10'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:54.950630 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-534fe5f2-c4f7-4507-b6b1-e598d538df10 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2dd38d98e8f042b69c60978ea8e7742b', 'request_id': 'req-534fe5f2-c4f7-4507-b6b1-e598d538df10'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:54.959602 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-534fe5f2-c4f7-4507-b6b1-e598d538df10 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2dd38d98e8f042b69c60978ea8e7742b', 'request_id': 'req-534fe5f2-c4f7-4507-b6b1-e598d538df10'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:54.983146 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-534fe5f2-c4f7-4507-b6b1-e598d538df10 admin admin] The token cache is being invalidate because project 2dd38d98e8f042b69c60978ea8e7742b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81434) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:35:54.983438 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-534fe5f2-c4f7-4507-b6b1-e598d538df10 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:54.984101 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-534fe5f2-c4f7-4507-b6b1-e598d538df10 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:35:54.987060 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1110/2219] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:35:54 2022] DELETE /identity/v3/projects/2dd38d98e8f042b69c60978ea8e7742b => generated 0 bytes in 177 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 21:35:54.995436 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a76dbc9-c21c-45bf-b1c8-97ef9fcdf44b None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:54.996250 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a76dbc9-c21c-45bf-b1c8-97ef9fcdf44b None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:54.996985 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a76dbc9-c21c-45bf-b1c8-97ef9fcdf44b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:55.041556 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-4a76dbc9-c21c-45bf-b1c8-97ef9fcdf44b None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:35:55.044939 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-4a76dbc9-c21c-45bf-b1c8-97ef9fcdf44b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:55.146966 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1110/2220] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:35:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:35:55.156028 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99fe25e2-e101-41f0-9ba7-ab4f484c7537 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:55.156325 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99fe25e2-e101-41f0-9ba7-ab4f484c7537 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:55.156596 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99fe25e2-e101-41f0-9ba7-ab4f484c7537 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:55.200037 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-99fe25e2-e101-41f0-9ba7-ab4f484c7537 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:35:55.201623 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-99fe25e2-e101-41f0-9ba7-ab4f484c7537 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:55.214653 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1111/2221] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:35:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 60 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:35:55.218787 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a76dbc9-c21c-45bf-b1c8-97ef9fcdf44b None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:55.232126 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a76dbc9-c21c-45bf-b1c8-97ef9fcdf44b None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:55.240358 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbb1697a-d875-4568-a0c0-cef475f597f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:55.241824 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbb1697a-d875-4568-a0c0-cef475f597f3 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:55.243096 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbb1697a-d875-4568-a0c0-cef475f597f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:55.243573 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbb1697a-d875-4568-a0c0-cef475f597f3 admin admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:55.244712 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbb1697a-d875-4568-a0c0-cef475f597f3 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:35:55.245640 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbb1697a-d875-4568-a0c0-cef475f597f3 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:55.248839 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbb1697a-d875-4568-a0c0-cef475f597f3 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:55.272194 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1111/2222] 10.222.0.90 () {58 vars in 1153 bytes} [Mon Aug 29 21:35:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:35:55.277923 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99fe25e2-e101-41f0-9ba7-ab4f484c7537 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:55.296296 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99fe25e2-e101-41f0-9ba7-ab4f484c7537 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:55.302063 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15404c92-791c-4660-a25f-3d6fdea813cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:55.302719 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15404c92-791c-4660-a25f-3d6fdea813cc admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:55.303004 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15404c92-791c-4660-a25f-3d6fdea813cc admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:55.303297 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15404c92-791c-4660-a25f-3d6fdea813cc admin admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:55.307088 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15404c92-791c-4660-a25f-3d6fdea813cc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:55.307088 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15404c92-791c-4660-a25f-3d6fdea813cc admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:55.349246 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1112/2223] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:35:55 2022] POST /identity/v3/projects => generated 383 bytes in 72 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:35:55.369208 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbb1697a-d875-4568-a0c0-cef475f597f3 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:55.383329 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbb1697a-d875-4568-a0c0-cef475f597f3 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:55.480098 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-118b8c65-cd65-47b7-80f1-e1774389139f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:55.481136 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-118b8c65-cd65-47b7-80f1-e1774389139f admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:55.481566 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-118b8c65-cd65-47b7-80f1-e1774389139f admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:55.481878 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-118b8c65-cd65-47b7-80f1-e1774389139f admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:55.482754 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-118b8c65-cd65-47b7-80f1-e1774389139f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:55.485209 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-118b8c65-cd65-47b7-80f1-e1774389139f admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:55.525123 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1112/2224] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:35:55 2022] POST /identity/v3/users => generated 292 bytes in 169 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:35:55.529336 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15404c92-791c-4660-a25f-3d6fdea813cc admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:55.545550 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15404c92-791c-4660-a25f-3d6fdea813cc admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:55.553946 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70a3d652-cb61-43f3-8dc5-d26c9ecc1481 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:55.554977 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a3d652-cb61-43f3-8dc5-d26c9ecc1481 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:55.555583 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a3d652-cb61-43f3-8dc5-d26c9ecc1481 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:55.556097 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a3d652-cb61-43f3-8dc5-d26c9ecc1481 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:55.556882 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70a3d652-cb61-43f3-8dc5-d26c9ecc1481 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:55.558948 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70a3d652-cb61-43f3-8dc5-d26c9ecc1481 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:55.574977 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1113/2225] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:35:55 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 29 21:35:55.579347 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-118b8c65-cd65-47b7-80f1-e1774389139f admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:55.593849 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-118b8c65-cd65-47b7-80f1-e1774389139f admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:55.599837 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0630f34c-bceb-474a-828e-2a256849664a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:55.600480 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0630f34c-bceb-474a-828e-2a256849664a admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:55.600753 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0630f34c-bceb-474a-828e-2a256849664a admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:55.601025 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0630f34c-bceb-474a-828e-2a256849664a admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:55.601679 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0630f34c-bceb-474a-828e-2a256849664a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:55.603372 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0630f34c-bceb-474a-828e-2a256849664a admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:55.618124 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1113/2226] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:35:55 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 29 21:35:55.622440 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70a3d652-cb61-43f3-8dc5-d26c9ecc1481 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:55.635813 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70a3d652-cb61-43f3-8dc5-d26c9ecc1481 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:55.641365 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1f7c073-73a8-4dff-9590-d3b6db4c6b8b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:55.642871 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1f7c073-73a8-4dff-9590-d3b6db4c6b8b admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:55.642871 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1f7c073-73a8-4dff-9590-d3b6db4c6b8b admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:55.643194 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1f7c073-73a8-4dff-9590-d3b6db4c6b8b admin admin] PATH_INFO: `/v3/projects/f3bcb7919eee471b8dd0a728fd6255a7/users/736ad47c636d4172b1a646437ac4feec/roles/02e242d5520740d4b77a5f4008c7eb57` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:55.665595 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1f7c073-73a8-4dff-9590-d3b6db4c6b8b admin admin] RBAC: Authorizing `identity:create_grant(project_id=f3bcb7919eee471b8dd0a728fd6255a7, user_id=736ad47c636d4172b1a646437ac4feec, role_id=02e242d5520740d4b77a5f4008c7eb57)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:55.670325 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1f7c073-73a8-4dff-9590-d3b6db4c6b8b admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:55.694694 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1114/2227] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:35:55 2022] PUT /identity/v3/projects/f3bcb7919eee471b8dd0a728fd6255a7/users/736ad47c636d4172b1a646437ac4feec/roles/02e242d5520740d4b77a5f4008c7eb57 => generated 0 bytes in 72 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:35:55.701547 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebec07c5-3e3c-4fec-b93e-46adb80d38d4 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:55.701938 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebec07c5-3e3c-4fec-b93e-46adb80d38d4 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:55.702213 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebec07c5-3e3c-4fec-b93e-46adb80d38d4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:55.738692 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-ebec07c5-3e3c-4fec-b93e-46adb80d38d4 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:35:55.742804 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-ebec07c5-3e3c-4fec-b93e-46adb80d38d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:55.761398 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:35:55.843310 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1114/2228] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:35:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:35:55.854038 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1f7c073-73a8-4dff-9590-d3b6db4c6b8b admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:55.858061 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-d1f7c073-73a8-4dff-9590-d3b6db4c6b8b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:55.934255 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1f7c073-73a8-4dff-9590-d3b6db4c6b8b admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:55.938702 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52382ca0-dca9-4f01-95d1-906157b60605 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:55.939887 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52382ca0-dca9-4f01-95d1-906157b60605 service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:55.939887 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52382ca0-dca9-4f01-95d1-906157b60605 service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:55.939887 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52382ca0-dca9-4f01-95d1-906157b60605 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:55.953211 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52382ca0-dca9-4f01-95d1-906157b60605 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:55.955277 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52382ca0-dca9-4f01-95d1-906157b60605 service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:55.973445 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1115/2229] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:35:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 116 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:35:57.182307 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25c50fe1-b355-4aba-9032-8eaf2a18d77d None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:57.183533 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25c50fe1-b355-4aba-9032-8eaf2a18d77d None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:57.183987 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25c50fe1-b355-4aba-9032-8eaf2a18d77d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:57.228925 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-25c50fe1-b355-4aba-9032-8eaf2a18d77d None None] MFA Rules not processed for user `736ad47c636d4172b1a646437ac4feec`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:35:57.230517 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-25c50fe1-b355-4aba-9032-8eaf2a18d77d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:57.349609 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1115/2230] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:35:57 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:35:57.364935 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52382ca0-dca9-4f01-95d1-906157b60605 service neutron] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:35:57.366513 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-52382ca0-dca9-4f01-95d1-906157b60605 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:57.454877 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52382ca0-dca9-4f01-95d1-906157b60605 service neutron] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:35:57.460279 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccb0f8f4-65cb-4142-9766-96c8b03f8a92 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dc226f7ddcb49ecacb51ce57fdfc11b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:35:57.461400 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccb0f8f4-65cb-4142-9766-96c8b03f8a92 service cinder] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:57.461840 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccb0f8f4-65cb-4142-9766-96c8b03f8a92 service cinder] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:57.462260 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccb0f8f4-65cb-4142-9766-96c8b03f8a92 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:57.480305 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccb0f8f4-65cb-4142-9766-96c8b03f8a92 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:35:57.483185 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccb0f8f4-65cb-4142-9766-96c8b03f8a92 service cinder] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:35:57.500027 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1116/2231] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:35:57 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 136 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:35:58.340509 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb6c44f5-2429-4be3-b0ed-eda085cb44eb None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:35:58.341127 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb6c44f5-2429-4be3-b0ed-eda085cb44eb None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:35:58.341308 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb6c44f5-2429-4be3-b0ed-eda085cb44eb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:35:58.370122 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-fb6c44f5-2429-4be3-b0ed-eda085cb44eb None None] MFA Rules not processed for user `bbbfb39899b04668ac1b7ff28d18e839`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:35:58.371391 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-fb6c44f5-2429-4be3-b0ed-eda085cb44eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:35:58.457055 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1116/2232] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:35:58 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:36:01.197005 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31474ebe-f198-496f-95b6-a439127b9a34 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:36:01.197556 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31474ebe-f198-496f-95b6-a439127b9a34 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:36:01.197556 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31474ebe-f198-496f-95b6-a439127b9a34 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:36:01.220166 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-31474ebe-f198-496f-95b6-a439127b9a34 None None] MFA Rules not processed for user `736ad47c636d4172b1a646437ac4feec`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:36:01.221417 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-31474ebe-f198-496f-95b6-a439127b9a34 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:36:01.228700 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1117/2233] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:36:01 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:36:03.480150 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac1bb0b5-4726-4b80-88bf-9e9f3b4bb4b9 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:36:03.480805 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac1bb0b5-4726-4b80-88bf-9e9f3b4bb4b9 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:36:03.480805 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac1bb0b5-4726-4b80-88bf-9e9f3b4bb4b9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:36:03.511056 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-ac1bb0b5-4726-4b80-88bf-9e9f3b4bb4b9 None None] MFA Rules not processed for user `39f161e61a514d94b6008bd006cda5ce`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:36:03.512638 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-ac1bb0b5-4726-4b80-88bf-9e9f3b4bb4b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:36:03.567578 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:36:03.581529 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1117/2234] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:36:03 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:36:06.291380 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31474ebe-f198-496f-95b6-a439127b9a34 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:36:06.293209 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-31474ebe-f198-496f-95b6-a439127b9a34 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:36:06.313317 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31474ebe-f198-496f-95b6-a439127b9a34 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:36:06.320735 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-077bce40-9d82-4f1c-a513-214ac9a43c6f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:36:06.321727 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-077bce40-9d82-4f1c-a513-214ac9a43c6f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:36:06.334380 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-077bce40-9d82-4f1c-a513-214ac9a43c6f admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:36:06.334663 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-077bce40-9d82-4f1c-a513-214ac9a43c6f admin admin] PATH_INFO: `/v3/users/1799b582f9a5462fbcd671b9415da8d3` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:36:06.338150 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-077bce40-9d82-4f1c-a513-214ac9a43c6f admin admin] RBAC: Authorizing `identity:delete_user(user_id=1799b582f9a5462fbcd671b9415da8d3)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:36:06.340198 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-077bce40-9d82-4f1c-a513-214ac9a43c6f admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:36:06.343342 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:DELETE=4,keystone:SELECT=93,keystone:INSERT=3 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:36:06.421567 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-077bce40-9d82-4f1c-a513-214ac9a43c6f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1799b582f9a5462fbcd671b9415da8d3'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:36:06.433917 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-077bce40-9d82-4f1c-a513-214ac9a43c6f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1799b582f9a5462fbcd671b9415da8d3'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:36:06.454642 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-077bce40-9d82-4f1c-a513-214ac9a43c6f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1799b582f9a5462fbcd671b9415da8d3'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:36:06.456246 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-077bce40-9d82-4f1c-a513-214ac9a43c6f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1799b582f9a5462fbcd671b9415da8d3'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:36:06.457570 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-077bce40-9d82-4f1c-a513-214ac9a43c6f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1799b582f9a5462fbcd671b9415da8d3'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:36:06.482741 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1118/2235] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:36:06 2022] DELETE /identity/v3/users/1799b582f9a5462fbcd671b9415da8d3 => generated 0 bytes in 192 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:36:06.490881 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd290ab-b9ba-4f10-8bdf-291cb49c9b43 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:36:06.490881 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd290ab-b9ba-4f10-8bdf-291cb49c9b43 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:36:06.490881 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd290ab-b9ba-4f10-8bdf-291cb49c9b43 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:36:06.512008 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-bcd290ab-b9ba-4f10-8bdf-291cb49c9b43 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:36:06.514108 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-bcd290ab-b9ba-4f10-8bdf-291cb49c9b43 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:36:06.600401 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1118/2236] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:36:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:36:06.613285 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-077bce40-9d82-4f1c-a513-214ac9a43c6f admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:36:06.614916 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-077bce40-9d82-4f1c-a513-214ac9a43c6f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:36:06.682111 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-077bce40-9d82-4f1c-a513-214ac9a43c6f admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:36:06.689100 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-669fbdf9-f5d8-418e-b7d1-89d24041164f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:36:06.690176 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-669fbdf9-f5d8-418e-b7d1-89d24041164f service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:36:06.690682 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-669fbdf9-f5d8-418e-b7d1-89d24041164f service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:36:06.691364 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-669fbdf9-f5d8-418e-b7d1-89d24041164f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:36:06.702960 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-669fbdf9-f5d8-418e-b7d1-89d24041164f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:36:06.704855 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-669fbdf9-f5d8-418e-b7d1-89d24041164f service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:36:06.716365 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1119/2237] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:36:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:36:07.111514 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcd290ab-b9ba-4f10-8bdf-291cb49c9b43 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:36:07.116074 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-bcd290ab-b9ba-4f10-8bdf-291cb49c9b43 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:36:07.132742 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcd290ab-b9ba-4f10-8bdf-291cb49c9b43 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:36:07.139930 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8e00a58-f6c1-4b9f-a85a-97550acb8f93 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:36:07.140910 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8e00a58-f6c1-4b9f-a85a-97550acb8f93 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:36:07.141824 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8e00a58-f6c1-4b9f-a85a-97550acb8f93 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:36:07.142096 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8e00a58-f6c1-4b9f-a85a-97550acb8f93 admin admin] PATH_INFO: `/v3/projects/33a79f7ecf224e03ae0e5bf5e12a7638` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:36:07.143322 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8e00a58-f6c1-4b9f-a85a-97550acb8f93 admin admin] RBAC: Authorizing `identity:delete_project(project_id=33a79f7ecf224e03ae0e5bf5e12a7638)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:36:07.145307 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8e00a58-f6c1-4b9f-a85a-97550acb8f93 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:36:07.200957 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-e8e00a58-f6c1-4b9f-a85a-97550acb8f93 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '33a79f7ecf224e03ae0e5bf5e12a7638', 'request_id': 'req-e8e00a58-f6c1-4b9f-a85a-97550acb8f93'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:36:07.209553 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-e8e00a58-f6c1-4b9f-a85a-97550acb8f93 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '33a79f7ecf224e03ae0e5bf5e12a7638', 'request_id': 'req-e8e00a58-f6c1-4b9f-a85a-97550acb8f93'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:36:07.217354 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-e8e00a58-f6c1-4b9f-a85a-97550acb8f93 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '33a79f7ecf224e03ae0e5bf5e12a7638', 'request_id': 'req-e8e00a58-f6c1-4b9f-a85a-97550acb8f93'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:36:07.234136 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-e8e00a58-f6c1-4b9f-a85a-97550acb8f93 admin admin] The token cache is being invalidate because project 33a79f7ecf224e03ae0e5bf5e12a7638 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:36:07.234569 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-e8e00a58-f6c1-4b9f-a85a-97550acb8f93 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:36:07.235410 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-e8e00a58-f6c1-4b9f-a85a-97550acb8f93 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:36:07.237016 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1119/2238] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:36:07 2022] DELETE /identity/v3/projects/33a79f7ecf224e03ae0e5bf5e12a7638 => generated 0 bytes in 127 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 21:36:07.245424 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c624e15-ba35-4429-a351-dce5c51b6ff3 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:36:07.245707 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c624e15-ba35-4429-a351-dce5c51b6ff3 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:36:07.246002 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c624e15-ba35-4429-a351-dce5c51b6ff3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:36:07.268466 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-3c624e15-ba35-4429-a351-dce5c51b6ff3 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:36:07.269892 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-3c624e15-ba35-4429-a351-dce5c51b6ff3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:36:07.355774 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1120/2239] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:36:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:36:07.365923 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29036ac9-af67-43d6-b96f-529fe19efa1f None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:36:07.366252 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29036ac9-af67-43d6-b96f-529fe19efa1f None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:36:07.366537 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29036ac9-af67-43d6-b96f-529fe19efa1f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:36:07.425175 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-29036ac9-af67-43d6-b96f-529fe19efa1f None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:36:07.432473 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-29036ac9-af67-43d6-b96f-529fe19efa1f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:36:07.450144 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1120/2240] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:36:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:36:07.453198 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c624e15-ba35-4429-a351-dce5c51b6ff3 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:36:07.472838 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c624e15-ba35-4429-a351-dce5c51b6ff3 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:36:07.487972 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41b14ad4-ee25-45a5-b9b2-ddbc51c08bab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:36:07.488615 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b14ad4-ee25-45a5-b9b2-ddbc51c08bab admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:36:07.488900 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b14ad4-ee25-45a5-b9b2-ddbc51c08bab admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:36:07.489173 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b14ad4-ee25-45a5-b9b2-ddbc51c08bab admin admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:36:07.490221 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41b14ad4-ee25-45a5-b9b2-ddbc51c08bab admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:36:07.490221 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41b14ad4-ee25-45a5-b9b2-ddbc51c08bab admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:36:07.491971 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41b14ad4-ee25-45a5-b9b2-ddbc51c08bab admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:36:07.512708 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29036ac9-af67-43d6-b96f-529fe19efa1f None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:36:07.514020 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1121/2241] 10.222.0.90 () {58 vars in 1153 bytes} [Mon Aug 29 21:36:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 61 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:36:07.538520 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29036ac9-af67-43d6-b96f-529fe19efa1f None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:36:07.544746 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92bdf5aa-5e88-45c0-8ca6-18921bc3a37d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:36:07.545345 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92bdf5aa-5e88-45c0-8ca6-18921bc3a37d admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:36:07.545627 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92bdf5aa-5e88-45c0-8ca6-18921bc3a37d admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:36:07.545962 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92bdf5aa-5e88-45c0-8ca6-18921bc3a37d admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:36:07.546611 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92bdf5aa-5e88-45c0-8ca6-18921bc3a37d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:36:07.550275 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92bdf5aa-5e88-45c0-8ca6-18921bc3a37d admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:36:07.583959 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1121/2242] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:36:07 2022] POST /identity/v3/projects => generated 383 bytes in 72 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:36:07.588288 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41b14ad4-ee25-45a5-b9b2-ddbc51c08bab admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:36:07.599181 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41b14ad4-ee25-45a5-b9b2-ddbc51c08bab admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:36:07.609181 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:36:07.668441 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aae3ec5f-bd0e-4d6f-ae3b-2b28ae739cd6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:36:07.669057 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aae3ec5f-bd0e-4d6f-ae3b-2b28ae739cd6 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:36:07.669996 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aae3ec5f-bd0e-4d6f-ae3b-2b28ae739cd6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:36:07.669996 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aae3ec5f-bd0e-4d6f-ae3b-2b28ae739cd6 admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:36:07.670423 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aae3ec5f-bd0e-4d6f-ae3b-2b28ae739cd6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:36:07.672434 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aae3ec5f-bd0e-4d6f-ae3b-2b28ae739cd6 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:36:07.694823 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1122/2243] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:36:07 2022] POST /identity/v3/users => generated 292 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:36:07.699029 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92bdf5aa-5e88-45c0-8ca6-18921bc3a37d admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:36:07.709942 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92bdf5aa-5e88-45c0-8ca6-18921bc3a37d admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:36:07.715752 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-618a2eab-cdaf-4035-a808-67e9765f51e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:36:07.716485 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-618a2eab-cdaf-4035-a808-67e9765f51e8 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:36:07.716831 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-618a2eab-cdaf-4035-a808-67e9765f51e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:36:07.717176 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-618a2eab-cdaf-4035-a808-67e9765f51e8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:36:07.717950 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-618a2eab-cdaf-4035-a808-67e9765f51e8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:36:07.719865 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-618a2eab-cdaf-4035-a808-67e9765f51e8 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:36:07.732312 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1122/2244] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:36:07 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:36:07.737391 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aae3ec5f-bd0e-4d6f-ae3b-2b28ae739cd6 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:36:07.752021 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aae3ec5f-bd0e-4d6f-ae3b-2b28ae739cd6 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:36:07.757675 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b60325f5-9fa9-46ec-9f95-1386f2025fd6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:36:07.758509 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b60325f5-9fa9-46ec-9f95-1386f2025fd6 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:36:07.758800 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b60325f5-9fa9-46ec-9f95-1386f2025fd6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:36:07.759809 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b60325f5-9fa9-46ec-9f95-1386f2025fd6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:36:07.759809 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b60325f5-9fa9-46ec-9f95-1386f2025fd6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:36:07.762192 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b60325f5-9fa9-46ec-9f95-1386f2025fd6 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:36:07.778734 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1123/2245] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:36:07 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 29 21:36:07.781828 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-618a2eab-cdaf-4035-a808-67e9765f51e8 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:36:07.793115 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-618a2eab-cdaf-4035-a808-67e9765f51e8 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:36:07.801396 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d8bd269-f645-488a-8b0d-08c531647ce7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:36:07.802884 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d8bd269-f645-488a-8b0d-08c531647ce7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:36:07.803222 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d8bd269-f645-488a-8b0d-08c531647ce7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:36:07.803543 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d8bd269-f645-488a-8b0d-08c531647ce7 admin admin] PATH_INFO: `/v3/projects/7a6c1eeec75e4bd980321f690e55c035/users/7eb2dcdd122243a38efd6a2f209b04d9/roles/02e242d5520740d4b77a5f4008c7eb57` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:36:07.824919 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d8bd269-f645-488a-8b0d-08c531647ce7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7a6c1eeec75e4bd980321f690e55c035, user_id=7eb2dcdd122243a38efd6a2f209b04d9, role_id=02e242d5520740d4b77a5f4008c7eb57)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:36:07.828022 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d8bd269-f645-488a-8b0d-08c531647ce7 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:36:07.853284 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1123/2246] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:36:07 2022] PUT /identity/v3/projects/7a6c1eeec75e4bd980321f690e55c035/users/7eb2dcdd122243a38efd6a2f209b04d9/roles/02e242d5520740d4b77a5f4008c7eb57 => generated 0 bytes in 72 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:36:07.865821 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48df3c6f-722b-47fd-9db9-f9e878e5ef48 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:36:07.866166 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48df3c6f-722b-47fd-9db9-f9e878e5ef48 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:36:07.866447 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48df3c6f-722b-47fd-9db9-f9e878e5ef48 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:36:07.908916 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-48df3c6f-722b-47fd-9db9-f9e878e5ef48 None None] MFA Rules not processed for user `7eb2dcdd122243a38efd6a2f209b04d9`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:36:07.912572 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-48df3c6f-722b-47fd-9db9-f9e878e5ef48 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:36:08.013208 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1124/2247] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:36:07 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:36:08.026303 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d8bd269-f645-488a-8b0d-08c531647ce7 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:36:08.028420 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-2d8bd269-f645-488a-8b0d-08c531647ce7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:36:08.046367 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:36:08.131484 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d8bd269-f645-488a-8b0d-08c531647ce7 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:36:08.136377 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be3c497f-0512-470e-be76-99950e09e4ad service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dc226f7ddcb49ecacb51ce57fdfc11b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:36:08.137007 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be3c497f-0512-470e-be76-99950e09e4ad service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:36:08.137376 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be3c497f-0512-470e-be76-99950e09e4ad service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:36:08.137660 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be3c497f-0512-470e-be76-99950e09e4ad service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:36:08.150040 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be3c497f-0512-470e-be76-99950e09e4ad service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:36:08.151511 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be3c497f-0512-470e-be76-99950e09e4ad service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:36:08.163674 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1124/2248] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:36:08 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 138 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:36:11.086895 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48df3c6f-722b-47fd-9db9-f9e878e5ef48 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:36:11.226724 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48df3c6f-722b-47fd-9db9-f9e878e5ef48 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:36:11.250777 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddeacdaa-10ee-4f41-bb29-b47eb9239331 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:36:11.252270 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddeacdaa-10ee-4f41-bb29-b47eb9239331 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:36:11.253028 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddeacdaa-10ee-4f41-bb29-b47eb9239331 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:36:11.253475 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddeacdaa-10ee-4f41-bb29-b47eb9239331 admin admin] PATH_INFO: `/v3/users/7eb2dcdd122243a38efd6a2f209b04d9` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:36:11.255447 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddeacdaa-10ee-4f41-bb29-b47eb9239331 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7eb2dcdd122243a38efd6a2f209b04d9)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:36:11.258466 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddeacdaa-10ee-4f41-bb29-b47eb9239331 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:36:11.303128 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=94,keystone:INSERT=3,keystone:DELETE=3 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:36:11.360917 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-ddeacdaa-10ee-4f41-bb29-b47eb9239331 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7eb2dcdd122243a38efd6a2f209b04d9'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:36:11.374320 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-ddeacdaa-10ee-4f41-bb29-b47eb9239331 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7eb2dcdd122243a38efd6a2f209b04d9'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:36:11.385747 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-ddeacdaa-10ee-4f41-bb29-b47eb9239331 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7eb2dcdd122243a38efd6a2f209b04d9'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:36:11.386639 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-ddeacdaa-10ee-4f41-bb29-b47eb9239331 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7eb2dcdd122243a38efd6a2f209b04d9'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:36:11.387578 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-ddeacdaa-10ee-4f41-bb29-b47eb9239331 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7eb2dcdd122243a38efd6a2f209b04d9'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:36:11.419052 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1125/2249] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:36:11 2022] DELETE /identity/v3/users/7eb2dcdd122243a38efd6a2f209b04d9 => generated 0 bytes in 333 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:36:11.424276 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5bbec56-ed78-456a-b553-3f02484593d7 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:36:11.424802 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5bbec56-ed78-456a-b553-3f02484593d7 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:36:11.424958 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5bbec56-ed78-456a-b553-3f02484593d7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:36:11.451988 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-b5bbec56-ed78-456a-b553-3f02484593d7 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:36:11.453252 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-b5bbec56-ed78-456a-b553-3f02484593d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:36:11.559421 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1125/2250] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:36:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:36:11.570902 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddeacdaa-10ee-4f41-bb29-b47eb9239331 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:36:11.572904 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-ddeacdaa-10ee-4f41-bb29-b47eb9239331 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:36:11.644931 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddeacdaa-10ee-4f41-bb29-b47eb9239331 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:36:11.648906 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aec515ef-71fc-49b9-8f52-56644935586c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:36:11.649556 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aec515ef-71fc-49b9-8f52-56644935586c service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:36:11.649865 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aec515ef-71fc-49b9-8f52-56644935586c service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:36:11.650147 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aec515ef-71fc-49b9-8f52-56644935586c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:36:11.662244 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aec515ef-71fc-49b9-8f52-56644935586c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:36:11.663735 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aec515ef-71fc-49b9-8f52-56644935586c service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:36:11.677472 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1126/2251] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:36:11 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:36:12.159966 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5bbec56-ed78-456a-b553-3f02484593d7 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:36:12.161959 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-b5bbec56-ed78-456a-b553-3f02484593d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:36:12.184922 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5bbec56-ed78-456a-b553-3f02484593d7 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:36:12.193097 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3079dba2-7161-4a03-aebb-5328a460ab1b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:36:12.194134 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3079dba2-7161-4a03-aebb-5328a460ab1b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:36:12.194453 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3079dba2-7161-4a03-aebb-5328a460ab1b admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:36:12.194621 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3079dba2-7161-4a03-aebb-5328a460ab1b admin admin] PATH_INFO: `/v3/projects/7a6c1eeec75e4bd980321f690e55c035` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:36:12.195715 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3079dba2-7161-4a03-aebb-5328a460ab1b admin admin] RBAC: Authorizing `identity:delete_project(project_id=7a6c1eeec75e4bd980321f690e55c035)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:36:12.197685 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3079dba2-7161-4a03-aebb-5328a460ab1b admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:36:12.280651 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-3079dba2-7161-4a03-aebb-5328a460ab1b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7a6c1eeec75e4bd980321f690e55c035', 'request_id': 'req-3079dba2-7161-4a03-aebb-5328a460ab1b'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:36:12.295018 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-3079dba2-7161-4a03-aebb-5328a460ab1b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7a6c1eeec75e4bd980321f690e55c035', 'request_id': 'req-3079dba2-7161-4a03-aebb-5328a460ab1b'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:36:12.307160 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-3079dba2-7161-4a03-aebb-5328a460ab1b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7a6c1eeec75e4bd980321f690e55c035', 'request_id': 'req-3079dba2-7161-4a03-aebb-5328a460ab1b'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:36:12.340153 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-3079dba2-7161-4a03-aebb-5328a460ab1b admin admin] The token cache is being invalidate because project 7a6c1eeec75e4bd980321f690e55c035 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:36:12.340493 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-3079dba2-7161-4a03-aebb-5328a460ab1b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:36:12.341310 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-3079dba2-7161-4a03-aebb-5328a460ab1b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:36:12.343014 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1126/2252] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:36:12 2022] DELETE /identity/v3/projects/7a6c1eeec75e4bd980321f690e55c035 => generated 0 bytes in 184 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:36:12.352425 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33940ecf-5760-470d-b7e3-235c41937b73 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:36:12.353390 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33940ecf-5760-470d-b7e3-235c41937b73 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:36:12.353957 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33940ecf-5760-470d-b7e3-235c41937b73 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:36:12.406096 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-33940ecf-5760-470d-b7e3-235c41937b73 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:36:12.411286 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-33940ecf-5760-470d-b7e3-235c41937b73 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:36:12.497441 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1127/2253] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:36:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 147 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:36:12.506563 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b8d968e-3205-4deb-b4d1-413591a3b86c None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:36:12.507118 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b8d968e-3205-4deb-b4d1-413591a3b86c None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:36:12.507118 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b8d968e-3205-4deb-b4d1-413591a3b86c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:36:12.529407 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-3b8d968e-3205-4deb-b4d1-413591a3b86c None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:36:12.530868 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-3b8d968e-3205-4deb-b4d1-413591a3b86c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:36:12.548997 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1127/2254] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:36:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:36:12.553144 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33940ecf-5760-470d-b7e3-235c41937b73 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:36:12.569874 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33940ecf-5760-470d-b7e3-235c41937b73 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:36:12.578291 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71dbdb02-4522-469b-84d9-8453c4792dee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:36:12.579308 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71dbdb02-4522-469b-84d9-8453c4792dee admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:36:12.579798 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71dbdb02-4522-469b-84d9-8453c4792dee admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:36:12.580107 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71dbdb02-4522-469b-84d9-8453c4792dee admin admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:36:12.581570 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71dbdb02-4522-469b-84d9-8453c4792dee admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:36:12.582051 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71dbdb02-4522-469b-84d9-8453c4792dee admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:36:12.584562 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71dbdb02-4522-469b-84d9-8453c4792dee admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:36:12.602664 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1128/2255] 10.222.0.90 () {58 vars in 1153 bytes} [Mon Aug 29 21:36:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:36:12.605751 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b8d968e-3205-4deb-b4d1-413591a3b86c None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:36:12.618988 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b8d968e-3205-4deb-b4d1-413591a3b86c None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:36:12.626092 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e19580e-2fa1-432c-af3c-f2fb3fcf6137 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:36:12.626817 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e19580e-2fa1-432c-af3c-f2fb3fcf6137 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:36:12.627160 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e19580e-2fa1-432c-af3c-f2fb3fcf6137 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:36:12.627513 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e19580e-2fa1-432c-af3c-f2fb3fcf6137 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:36:12.628181 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e19580e-2fa1-432c-af3c-f2fb3fcf6137 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:36:12.630210 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e19580e-2fa1-432c-af3c-f2fb3fcf6137 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:36:12.675422 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1128/2256] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:36:12 2022] POST /identity/v3/projects => generated 371 bytes in 70 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:36:12.679680 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71dbdb02-4522-469b-84d9-8453c4792dee admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:36:12.700228 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71dbdb02-4522-469b-84d9-8453c4792dee admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:36:12.739747 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:DELETE=7,keystone:SELECT=92,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:36:12.783657 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fc822a3-e314-47c7-904b-58bd376c0664 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:36:12.784573 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc822a3-e314-47c7-904b-58bd376c0664 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:36:12.784690 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc822a3-e314-47c7-904b-58bd376c0664 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:36:12.785485 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc822a3-e314-47c7-904b-58bd376c0664 admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:36:12.786580 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fc822a3-e314-47c7-904b-58bd376c0664 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:36:12.790764 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fc822a3-e314-47c7-904b-58bd376c0664 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:36:12.819834 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1129/2257] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:36:12 2022] POST /identity/v3/users => generated 286 bytes in 141 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 21:36:12.825092 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e19580e-2fa1-432c-af3c-f2fb3fcf6137 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:36:12.838811 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e19580e-2fa1-432c-af3c-f2fb3fcf6137 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:36:12.890820 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-420102fa-1119-4ca5-b4e5-ffd331dd15a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:36:12.892555 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-420102fa-1119-4ca5-b4e5-ffd331dd15a7 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:36:12.892555 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-420102fa-1119-4ca5-b4e5-ffd331dd15a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:36:12.892555 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-420102fa-1119-4ca5-b4e5-ffd331dd15a7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:36:12.892918 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-420102fa-1119-4ca5-b4e5-ffd331dd15a7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:36:12.894715 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-420102fa-1119-4ca5-b4e5-ffd331dd15a7 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:36:12.901489 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:36:12.910294 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1129/2258] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:36:12 2022] GET /identity/v3/roles => generated 1420 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:36:12.920468 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fc822a3-e314-47c7-904b-58bd376c0664 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:36:12.950566 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fc822a3-e314-47c7-904b-58bd376c0664 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:36:12.959892 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72922cb1-485a-46ea-b467-b82da0f2adee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:36:12.960782 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72922cb1-485a-46ea-b467-b82da0f2adee admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:36:12.961069 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72922cb1-485a-46ea-b467-b82da0f2adee admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:36:12.961341 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72922cb1-485a-46ea-b467-b82da0f2adee admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:36:12.962352 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72922cb1-485a-46ea-b467-b82da0f2adee admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:36:12.964767 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72922cb1-485a-46ea-b467-b82da0f2adee admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:36:12.988397 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1130/2259] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:36:12 2022] GET /identity/v3/roles => generated 1420 bytes in 73 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 29 21:36:12.996185 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-420102fa-1119-4ca5-b4e5-ffd331dd15a7 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:36:13.011760 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-420102fa-1119-4ca5-b4e5-ffd331dd15a7 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:36:13.018711 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3da30f6a-d26a-4fbf-8d52-2126071be87d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:36:13.019662 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3da30f6a-d26a-4fbf-8d52-2126071be87d admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:36:13.020091 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3da30f6a-d26a-4fbf-8d52-2126071be87d admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:36:13.020532 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3da30f6a-d26a-4fbf-8d52-2126071be87d admin admin] PATH_INFO: `/v3/projects/69d7f124802b450eae69db4d42b62d83/users/3376818708e443928951fd5f517c9d83/roles/02e242d5520740d4b77a5f4008c7eb57` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:36:13.038366 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3da30f6a-d26a-4fbf-8d52-2126071be87d admin admin] RBAC: Authorizing `identity:create_grant(project_id=69d7f124802b450eae69db4d42b62d83, user_id=3376818708e443928951fd5f517c9d83, role_id=02e242d5520740d4b77a5f4008c7eb57)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:36:13.041332 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3da30f6a-d26a-4fbf-8d52-2126071be87d admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:36:13.066119 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbf7e0cd-1d2f-489a-90b4-06513dad8cca None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:36:13.068094 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbf7e0cd-1d2f-489a-90b4-06513dad8cca None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:36:13.068398 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbf7e0cd-1d2f-489a-90b4-06513dad8cca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:36:13.071101 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1130/2260] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:36:12 2022] PUT /identity/v3/projects/69d7f124802b450eae69db4d42b62d83/users/3376818708e443928951fd5f517c9d83/roles/02e242d5520740d4b77a5f4008c7eb57 => generated 0 bytes in 64 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:36:13.102275 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-fbf7e0cd-1d2f-489a-90b4-06513dad8cca None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:36:13.104351 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-fbf7e0cd-1d2f-489a-90b4-06513dad8cca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:36:13.249036 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3da30f6a-d26a-4fbf-8d52-2126071be87d admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:36:13.250937 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1131/2261] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:36:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:36:13.269325 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-3da30f6a-d26a-4fbf-8d52-2126071be87d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:36:13.398090 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3da30f6a-d26a-4fbf-8d52-2126071be87d admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:36:13.403005 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e78ba65b-fb3c-4920-982f-1bf16d88dc0b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:36:13.403758 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e78ba65b-fb3c-4920-982f-1bf16d88dc0b service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:36:13.404216 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e78ba65b-fb3c-4920-982f-1bf16d88dc0b service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:36:13.404565 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e78ba65b-fb3c-4920-982f-1bf16d88dc0b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:36:13.417302 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e78ba65b-fb3c-4920-982f-1bf16d88dc0b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:36:13.418992 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e78ba65b-fb3c-4920-982f-1bf16d88dc0b service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:36:13.434355 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1131/2262] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:36:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 186 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 21:36:18.557575 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbf7e0cd-1d2f-489a-90b4-06513dad8cca None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:36:18.560084 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-fbf7e0cd-1d2f-489a-90b4-06513dad8cca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:36:18.576860 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbf7e0cd-1d2f-489a-90b4-06513dad8cca None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:36:18.585728 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a3b74a9-a085-4c1b-9387-ed452e41a8df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:36:18.586506 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a3b74a9-a085-4c1b-9387-ed452e41a8df admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:36:18.586899 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a3b74a9-a085-4c1b-9387-ed452e41a8df admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:36:18.587255 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a3b74a9-a085-4c1b-9387-ed452e41a8df admin admin] PATH_INFO: `/v3/users/6e285737d3d646389792cc57b912f475` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:36:18.588702 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a3b74a9-a085-4c1b-9387-ed452e41a8df admin admin] RBAC: Authorizing `identity:delete_user(user_id=6e285737d3d646389792cc57b912f475)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:36:18.591260 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a3b74a9-a085-4c1b-9387-ed452e41a8df admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:36:18.655489 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-4a3b74a9-a085-4c1b-9387-ed452e41a8df admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6e285737d3d646389792cc57b912f475'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:36:18.665630 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-4a3b74a9-a085-4c1b-9387-ed452e41a8df admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6e285737d3d646389792cc57b912f475'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:36:18.678160 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-4a3b74a9-a085-4c1b-9387-ed452e41a8df admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6e285737d3d646389792cc57b912f475'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:36:18.679678 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-4a3b74a9-a085-4c1b-9387-ed452e41a8df admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6e285737d3d646389792cc57b912f475'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:36:18.680911 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-4a3b74a9-a085-4c1b-9387-ed452e41a8df admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6e285737d3d646389792cc57b912f475'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:36:18.716372 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1132/2263] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:36:18 2022] DELETE /identity/v3/users/6e285737d3d646389792cc57b912f475 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:36:18.718260 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-576231b5-360c-40ed-abc5-480f35478a91 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:36:18.719413 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-576231b5-360c-40ed-abc5-480f35478a91 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:36:18.720155 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-576231b5-360c-40ed-abc5-480f35478a91 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:36:18.744335 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-576231b5-360c-40ed-abc5-480f35478a91 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:36:18.745726 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-576231b5-360c-40ed-abc5-480f35478a91 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:36:18.879427 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1132/2264] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:36:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:36:18.894189 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a3b74a9-a085-4c1b-9387-ed452e41a8df admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:36:18.894189 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-4a3b74a9-a085-4c1b-9387-ed452e41a8df admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:36:18.935570 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=86,keystone:INSERT=4,keystone:DELETE=10 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:36:19.015151 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a3b74a9-a085-4c1b-9387-ed452e41a8df admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:36:19.023507 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e458c7e-33e7-4cd1-9e7e-6a4224ff10db service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:36:19.024480 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e458c7e-33e7-4cd1-9e7e-6a4224ff10db service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:36:19.025120 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e458c7e-33e7-4cd1-9e7e-6a4224ff10db service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:36:19.025529 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e458c7e-33e7-4cd1-9e7e-6a4224ff10db service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:36:19.044596 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e458c7e-33e7-4cd1-9e7e-6a4224ff10db service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:36:19.047234 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e458c7e-33e7-4cd1-9e7e-6a4224ff10db service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:36:19.065015 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1133/2265] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:36:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 21:36:20.191596 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-576231b5-360c-40ed-abc5-480f35478a91 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:36:20.193286 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-576231b5-360c-40ed-abc5-480f35478a91 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:36:20.222577 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-576231b5-360c-40ed-abc5-480f35478a91 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:36:20.244137 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c01923cd-ffcb-4cb0-9c70-ca9d71d54a29 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:36:20.245028 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c01923cd-ffcb-4cb0-9c70-ca9d71d54a29 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:36:20.245509 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c01923cd-ffcb-4cb0-9c70-ca9d71d54a29 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:36:20.245810 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c01923cd-ffcb-4cb0-9c70-ca9d71d54a29 admin admin] PATH_INFO: `/v3/projects/1faf244461314ecfab36d6f3d23c820c` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:36:20.247724 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c01923cd-ffcb-4cb0-9c70-ca9d71d54a29 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1faf244461314ecfab36d6f3d23c820c)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:36:20.253328 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c01923cd-ffcb-4cb0-9c70-ca9d71d54a29 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:36:20.351397 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-c01923cd-ffcb-4cb0-9c70-ca9d71d54a29 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1faf244461314ecfab36d6f3d23c820c', 'request_id': 'req-c01923cd-ffcb-4cb0-9c70-ca9d71d54a29'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:36:20.360900 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-c01923cd-ffcb-4cb0-9c70-ca9d71d54a29 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1faf244461314ecfab36d6f3d23c820c', 'request_id': 'req-c01923cd-ffcb-4cb0-9c70-ca9d71d54a29'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:36:20.369677 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-c01923cd-ffcb-4cb0-9c70-ca9d71d54a29 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1faf244461314ecfab36d6f3d23c820c', 'request_id': 'req-c01923cd-ffcb-4cb0-9c70-ca9d71d54a29'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:36:20.397915 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-c01923cd-ffcb-4cb0-9c70-ca9d71d54a29 admin admin] The token cache is being invalidate because project 1faf244461314ecfab36d6f3d23c820c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:36:20.398345 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-c01923cd-ffcb-4cb0-9c70-ca9d71d54a29 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:36:20.399230 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-c01923cd-ffcb-4cb0-9c70-ca9d71d54a29 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:36:20.401104 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1133/2266] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:36:20 2022] DELETE /identity/v3/projects/1faf244461314ecfab36d6f3d23c820c => generated 0 bytes in 210 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:36:20.407180 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e1b271d-f326-4f04-9ccf-2624d62f8a8a None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:36:20.407911 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e1b271d-f326-4f04-9ccf-2624d62f8a8a None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:36:20.407911 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e1b271d-f326-4f04-9ccf-2624d62f8a8a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:36:20.429974 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-6e1b271d-f326-4f04-9ccf-2624d62f8a8a None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:36:20.431244 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-6e1b271d-f326-4f04-9ccf-2624d62f8a8a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:36:20.548830 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1134/2267] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:36:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:36:20.560530 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ca07872-77cb-4afa-9db7-3ab7a5efaf25 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:36:20.560963 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ca07872-77cb-4afa-9db7-3ab7a5efaf25 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:36:20.561490 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ca07872-77cb-4afa-9db7-3ab7a5efaf25 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:36:20.586716 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:36:20.620574 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-9ca07872-77cb-4afa-9db7-3ab7a5efaf25 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:36:20.622117 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-9ca07872-77cb-4afa-9db7-3ab7a5efaf25 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:36:20.637969 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1134/2268] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:36:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:36:20.643054 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e1b271d-f326-4f04-9ccf-2624d62f8a8a None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:36:20.655789 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e1b271d-f326-4f04-9ccf-2624d62f8a8a None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:36:20.661605 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44122205-bb42-4dc2-b641-c25b95bc32b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:36:20.662500 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44122205-bb42-4dc2-b641-c25b95bc32b6 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:36:20.662783 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44122205-bb42-4dc2-b641-c25b95bc32b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:36:20.663055 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44122205-bb42-4dc2-b641-c25b95bc32b6 admin admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:36:20.664368 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44122205-bb42-4dc2-b641-c25b95bc32b6 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:36:20.664801 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44122205-bb42-4dc2-b641-c25b95bc32b6 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:36:20.666475 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44122205-bb42-4dc2-b641-c25b95bc32b6 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:36:20.699104 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1135/2269] 10.222.0.90 () {58 vars in 1153 bytes} [Mon Aug 29 21:36:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:36:20.700642 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ca07872-77cb-4afa-9db7-3ab7a5efaf25 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:36:20.715141 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ca07872-77cb-4afa-9db7-3ab7a5efaf25 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:36:20.742776 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19d77ad5-2556-4611-960b-3f475597c083 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:36:20.742776 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d77ad5-2556-4611-960b-3f475597c083 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:36:20.742776 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d77ad5-2556-4611-960b-3f475597c083 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:36:20.742776 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d77ad5-2556-4611-960b-3f475597c083 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:36:20.744022 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19d77ad5-2556-4611-960b-3f475597c083 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:36:20.745074 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19d77ad5-2556-4611-960b-3f475597c083 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:36:20.838785 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1135/2270] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:36:20 2022] POST /identity/v3/projects => generated 383 bytes in 139 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:36:20.843908 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44122205-bb42-4dc2-b641-c25b95bc32b6 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:36:20.861512 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44122205-bb42-4dc2-b641-c25b95bc32b6 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:36:20.989899 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c80a2388-72f9-4d95-891d-99a6f62645bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:36:20.990671 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c80a2388-72f9-4d95-891d-99a6f62645bf admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:36:20.991056 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c80a2388-72f9-4d95-891d-99a6f62645bf admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:36:20.991436 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c80a2388-72f9-4d95-891d-99a6f62645bf admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:36:20.992248 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c80a2388-72f9-4d95-891d-99a6f62645bf admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:36:20.994603 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c80a2388-72f9-4d95-891d-99a6f62645bf admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:36:21.028306 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1136/2271] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:36:20 2022] POST /identity/v3/users => generated 292 bytes in 185 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:36:21.032575 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19d77ad5-2556-4611-960b-3f475597c083 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:36:21.046338 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19d77ad5-2556-4611-960b-3f475597c083 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:36:21.053094 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f9e3afb-41c8-4ff9-9f8a-51da179e4865 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:36:21.053519 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f9e3afb-41c8-4ff9-9f8a-51da179e4865 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:36:21.053941 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f9e3afb-41c8-4ff9-9f8a-51da179e4865 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:36:21.056269 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-270c5645-1f5a-4747-bf4c-6abcc187ce64 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:36:21.057146 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-270c5645-1f5a-4747-bf4c-6abcc187ce64 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:36:21.057617 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-270c5645-1f5a-4747-bf4c-6abcc187ce64 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:36:21.058079 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-270c5645-1f5a-4747-bf4c-6abcc187ce64 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:36:21.058902 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-270c5645-1f5a-4747-bf4c-6abcc187ce64 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:36:21.060842 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-270c5645-1f5a-4747-bf4c-6abcc187ce64 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:36:21.076322 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1136/2272] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:36:21 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 29 21:36:21.083150 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-270c5645-1f5a-4747-bf4c-6abcc187ce64 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:36:21.098834 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-0f9e3afb-41c8-4ff9-9f8a-51da179e4865 None None] MFA Rules not processed for user `3376818708e443928951fd5f517c9d83`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:36:21.101131 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-0f9e3afb-41c8-4ff9-9f8a-51da179e4865 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:36:21.102486 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-270c5645-1f5a-4747-bf4c-6abcc187ce64 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:36:21.108600 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71698961-a610-4370-8606-22fbf79f0090 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:36:21.109571 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71698961-a610-4370-8606-22fbf79f0090 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:36:21.110056 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71698961-a610-4370-8606-22fbf79f0090 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:36:21.110506 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71698961-a610-4370-8606-22fbf79f0090 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:36:21.111512 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71698961-a610-4370-8606-22fbf79f0090 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:36:21.113512 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71698961-a610-4370-8606-22fbf79f0090 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:36:21.128045 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1137/2273] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:36:21 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:36:21.132775 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71698961-a610-4370-8606-22fbf79f0090 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:36:21.146743 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:36:21.152688 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71698961-a610-4370-8606-22fbf79f0090 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:36:21.165906 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e769a4a7-80e0-405d-86d6-f9d3825d264e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:36:21.166929 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e769a4a7-80e0-405d-86d6-f9d3825d264e admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:36:21.167318 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e769a4a7-80e0-405d-86d6-f9d3825d264e admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:36:21.167714 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e769a4a7-80e0-405d-86d6-f9d3825d264e admin admin] PATH_INFO: `/v3/projects/df74318c1e404179bbb8533b5909deb8/users/fdba2832a619433eb7e607bb8bf45c59/roles/02e242d5520740d4b77a5f4008c7eb57` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:36:21.193432 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e769a4a7-80e0-405d-86d6-f9d3825d264e admin admin] RBAC: Authorizing `identity:create_grant(project_id=df74318c1e404179bbb8533b5909deb8, user_id=fdba2832a619433eb7e607bb8bf45c59, role_id=02e242d5520740d4b77a5f4008c7eb57)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:36:21.199765 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e769a4a7-80e0-405d-86d6-f9d3825d264e admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:36:21.218865 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1137/2274] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:36:21 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 21:36:21.220758 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f9e3afb-41c8-4ff9-9f8a-51da179e4865 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:36:21.223373 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-0f9e3afb-41c8-4ff9-9f8a-51da179e4865 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:36:21.259288 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f9e3afb-41c8-4ff9-9f8a-51da179e4865 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:36:21.259559 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1138/2275] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:36:21 2022] PUT /identity/v3/projects/df74318c1e404179bbb8533b5909deb8/users/fdba2832a619433eb7e607bb8bf45c59/roles/02e242d5520740d4b77a5f4008c7eb57 => generated 0 bytes in 125 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 21:36:21.265832 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98fd1434-b840-45c3-8b30-5ac7bac0a119 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:36:21.266733 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98fd1434-b840-45c3-8b30-5ac7bac0a119 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:36:21.267335 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98fd1434-b840-45c3-8b30-5ac7bac0a119 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:36:21.307284 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-98fd1434-b840-45c3-8b30-5ac7bac0a119 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:36:21.309811 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-98fd1434-b840-45c3-8b30-5ac7bac0a119 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:36:21.367848 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a47aeb62-4ae5-429f-a9dd-167cdebcc47d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:36:21.368795 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a47aeb62-4ae5-429f-a9dd-167cdebcc47d admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:36:21.369313 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a47aeb62-4ae5-429f-a9dd-167cdebcc47d admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:36:21.369754 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a47aeb62-4ae5-429f-a9dd-167cdebcc47d admin admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:36:21.370699 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a47aeb62-4ae5-429f-a9dd-167cdebcc47d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:36:21.375513 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a47aeb62-4ae5-429f-a9dd-167cdebcc47d admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:36:21.448511 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1139/2276] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:36:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:36:21.455222 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1138/2277] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:36:21 2022] POST /identity/v3/projects => generated 373 bytes in 232 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:36:21.457058 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98fd1434-b840-45c3-8b30-5ac7bac0a119 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:36:21.473057 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a47aeb62-4ae5-429f-a9dd-167cdebcc47d admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:36:21.473632 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98fd1434-b840-45c3-8b30-5ac7bac0a119 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:36:21.480548 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-a47aeb62-4ae5-429f-a9dd-167cdebcc47d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:36:21.567392 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a47aeb62-4ae5-429f-a9dd-167cdebcc47d admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:36:21.576169 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2541f6a3-b2e8-4bf5-96fd-06dc9904dd15 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:36:21.576945 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2541f6a3-b2e8-4bf5-96fd-06dc9904dd15 service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:36:21.577219 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2541f6a3-b2e8-4bf5-96fd-06dc9904dd15 service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:36:21.577498 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2541f6a3-b2e8-4bf5-96fd-06dc9904dd15 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:36:21.601261 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45b86d32-d8ee-49ba-a1e1-f22dc08e32bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:36:21.602098 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45b86d32-d8ee-49ba-a1e1-f22dc08e32bc admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:36:21.602610 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45b86d32-d8ee-49ba-a1e1-f22dc08e32bc admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:36:21.603465 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45b86d32-d8ee-49ba-a1e1-f22dc08e32bc admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:36:21.603574 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45b86d32-d8ee-49ba-a1e1-f22dc08e32bc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:36:21.605702 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45b86d32-d8ee-49ba-a1e1-f22dc08e32bc admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:36:21.643329 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1140/2278] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:36:21 2022] POST /identity/v3/users => generated 293 bytes in 187 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 21:36:21.644822 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45b86d32-d8ee-49ba-a1e1-f22dc08e32bc admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:36:21.655662 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45b86d32-d8ee-49ba-a1e1-f22dc08e32bc admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:36:21.660994 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e0ac702-2de3-4b3a-b785-9ecd6ceb831d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:36:21.661729 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e0ac702-2de3-4b3a-b785-9ecd6ceb831d admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:36:21.662101 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e0ac702-2de3-4b3a-b785-9ecd6ceb831d admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:36:21.662444 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e0ac702-2de3-4b3a-b785-9ecd6ceb831d admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:36:21.663166 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e0ac702-2de3-4b3a-b785-9ecd6ceb831d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:36:21.664963 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e0ac702-2de3-4b3a-b785-9ecd6ceb831d admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:36:21.675329 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2541f6a3-b2e8-4bf5-96fd-06dc9904dd15 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:36:21.677235 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2541f6a3-b2e8-4bf5-96fd-06dc9904dd15 service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:36:21.678937 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1141/2279] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:36:21 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:36:21.683547 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e0ac702-2de3-4b3a-b785-9ecd6ceb831d admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:36:21.690205 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1139/2280] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:36:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 217 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:36:21.695272 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e0ac702-2de3-4b3a-b785-9ecd6ceb831d admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:36:21.701262 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0d1c054-349f-4050-bb07-eaf711227d22 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:36:21.702160 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0d1c054-349f-4050-bb07-eaf711227d22 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:36:21.702443 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0d1c054-349f-4050-bb07-eaf711227d22 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:36:21.702743 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0d1c054-349f-4050-bb07-eaf711227d22 admin admin] PATH_INFO: `/v3/projects/c77def5a174a4c45a37f0eea07e62368/users/5c64b71b38a24a9d8242e8cae089410a/roles/a6e562cdc96c4be691d638990f8403fd` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:36:21.705646 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:36:21.729249 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0d1c054-349f-4050-bb07-eaf711227d22 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c77def5a174a4c45a37f0eea07e62368, user_id=5c64b71b38a24a9d8242e8cae089410a, role_id=a6e562cdc96c4be691d638990f8403fd)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:36:21.732656 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0d1c054-349f-4050-bb07-eaf711227d22 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:36:21.752471 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1142/2281] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:36:21 2022] PUT /identity/v3/projects/c77def5a174a4c45a37f0eea07e62368/users/5c64b71b38a24a9d8242e8cae089410a/roles/a6e562cdc96c4be691d638990f8403fd => generated 0 bytes in 69 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:36:21.756214 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2541f6a3-b2e8-4bf5-96fd-06dc9904dd15 service neutron] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:36:21.762794 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:36:21.771872 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2541f6a3-b2e8-4bf5-96fd-06dc9904dd15 service neutron] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:36:21.913272 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-506e39f3-3f18-4713-b6f6-fdadc7bab27f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:36:21.914425 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-506e39f3-3f18-4713-b6f6-fdadc7bab27f admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:36:21.914908 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-506e39f3-3f18-4713-b6f6-fdadc7bab27f admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:36:21.915689 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-506e39f3-3f18-4713-b6f6-fdadc7bab27f admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:36:21.916620 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-506e39f3-3f18-4713-b6f6-fdadc7bab27f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:36:21.920088 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-506e39f3-3f18-4713-b6f6-fdadc7bab27f admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:36:21.940830 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1140/2282] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:36:21 2022] GET /identity/v3/roles => generated 1420 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:36:21.942833 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0d1c054-349f-4050-bb07-eaf711227d22 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:36:21.956464 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0d1c054-349f-4050-bb07-eaf711227d22 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:36:21.962567 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5aca3a9-240f-4976-9b1b-143277d54b3c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:36:21.963650 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5aca3a9-240f-4976-9b1b-143277d54b3c admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:36:21.963650 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5aca3a9-240f-4976-9b1b-143277d54b3c admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:36:21.963932 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5aca3a9-240f-4976-9b1b-143277d54b3c admin admin] PATH_INFO: `/v3/projects/c77def5a174a4c45a37f0eea07e62368/users/5c64b71b38a24a9d8242e8cae089410a/roles/a6e562cdc96c4be691d638990f8403fd` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:36:21.966529 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5aca3a9-240f-4976-9b1b-143277d54b3c admin admin] RBAC: Authorizing `identity:create_grant(project_id=c77def5a174a4c45a37f0eea07e62368, user_id=5c64b71b38a24a9d8242e8cae089410a, role_id=a6e562cdc96c4be691d638990f8403fd)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:36:21.970183 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5aca3a9-240f-4976-9b1b-143277d54b3c admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:36:22.006379 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1143/2283] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:36:21 2022] PUT /identity/v3/projects/c77def5a174a4c45a37f0eea07e62368/users/5c64b71b38a24a9d8242e8cae089410a/roles/a6e562cdc96c4be691d638990f8403fd => generated 0 bytes in 62 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 21:36:29.077335 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-758d4c46-b790-4331-b77e-2f5b28ae34bc None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:36:29.079139 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-758d4c46-b790-4331-b77e-2f5b28ae34bc None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:36:29.079865 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-758d4c46-b790-4331-b77e-2f5b28ae34bc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:36:29.189652 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-758d4c46-b790-4331-b77e-2f5b28ae34bc None None] MFA Rules not processed for user `fdba2832a619433eb7e607bb8bf45c59`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:36:29.191877 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-758d4c46-b790-4331-b77e-2f5b28ae34bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:36:29.426299 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1141/2284] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:36:29 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 362 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:36:29.432006 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5aca3a9-240f-4976-9b1b-143277d54b3c admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:36:29.579597 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5aca3a9-240f-4976-9b1b-143277d54b3c admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:36:29.602311 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e133388-5b46-46f1-91be-808c9a7f190d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:36:29.603564 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e133388-5b46-46f1-91be-808c9a7f190d admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:36:29.603859 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e133388-5b46-46f1-91be-808c9a7f190d admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:36:29.604146 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e133388-5b46-46f1-91be-808c9a7f190d admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:36:29.607686 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e133388-5b46-46f1-91be-808c9a7f190d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:36:29.611630 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e133388-5b46-46f1-91be-808c9a7f190d admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:36:29.673467 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1144/2285] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:36:29 2022] POST /identity/v3/projects => generated 383 bytes in 239 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:36:29.675202 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-758d4c46-b790-4331-b77e-2f5b28ae34bc None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:36:29.694066 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-758d4c46-b790-4331-b77e-2f5b28ae34bc None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:36:29.809567 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7168cec-6227-49a3-b0c8-f438a475b21a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:36:29.810929 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7168cec-6227-49a3-b0c8-f438a475b21a admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:36:29.811588 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7168cec-6227-49a3-b0c8-f438a475b21a admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:36:29.812389 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7168cec-6227-49a3-b0c8-f438a475b21a admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:36:29.813287 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7168cec-6227-49a3-b0c8-f438a475b21a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:36:29.816894 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7168cec-6227-49a3-b0c8-f438a475b21a admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:36:29.876871 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1142/2286] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:36:29 2022] POST /identity/v3/users => generated 298 bytes in 203 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:36:29.881621 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e133388-5b46-46f1-91be-808c9a7f190d admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:36:29.894800 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e133388-5b46-46f1-91be-808c9a7f190d admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:36:29.915196 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae0eb199-9a19-4de3-933d-38e27ee4fdb8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:36:29.915851 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae0eb199-9a19-4de3-933d-38e27ee4fdb8 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:36:29.916136 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae0eb199-9a19-4de3-933d-38e27ee4fdb8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:36:29.916415 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae0eb199-9a19-4de3-933d-38e27ee4fdb8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:36:29.917669 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae0eb199-9a19-4de3-933d-38e27ee4fdb8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:36:29.918873 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae0eb199-9a19-4de3-933d-38e27ee4fdb8 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:36:29.953067 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1145/2287] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:36:29 2022] GET /identity/v3/roles => generated 1420 bytes in 71 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:36:29.955985 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7168cec-6227-49a3-b0c8-f438a475b21a admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:36:29.992982 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7168cec-6227-49a3-b0c8-f438a475b21a admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:36:30.016353 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c008159e-98e6-4878-addb-bfeb5cf089ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:36:30.017403 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c008159e-98e6-4878-addb-bfeb5cf089ca admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:36:30.018167 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c008159e-98e6-4878-addb-bfeb5cf089ca admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:36:30.018341 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c008159e-98e6-4878-addb-bfeb5cf089ca admin admin] PATH_INFO: `/v3/projects/0c147e575342488f850408670d8838c2/users/8fc162f0d039495da2367c5189f91028/roles/a6e562cdc96c4be691d638990f8403fd` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:36:30.089968 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c008159e-98e6-4878-addb-bfeb5cf089ca admin admin] RBAC: Authorizing `identity:create_grant(project_id=0c147e575342488f850408670d8838c2, user_id=8fc162f0d039495da2367c5189f91028, role_id=a6e562cdc96c4be691d638990f8403fd)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:36:30.095344 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c008159e-98e6-4878-addb-bfeb5cf089ca admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:36:30.114426 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:36:30.134514 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1143/2288] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:36:29 2022] PUT /identity/v3/projects/0c147e575342488f850408670d8838c2/users/8fc162f0d039495da2367c5189f91028/roles/a6e562cdc96c4be691d638990f8403fd => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:36:30.152723 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae0eb199-9a19-4de3-933d-38e27ee4fdb8 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:36:30.171737 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae0eb199-9a19-4de3-933d-38e27ee4fdb8 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:36:30.284720 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c92b9ac5-a13e-4854-8748-99dd39f8b8eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:36:30.285602 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c92b9ac5-a13e-4854-8748-99dd39f8b8eb admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:36:30.286033 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c92b9ac5-a13e-4854-8748-99dd39f8b8eb admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:36:30.286409 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c92b9ac5-a13e-4854-8748-99dd39f8b8eb admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:36:30.287183 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c92b9ac5-a13e-4854-8748-99dd39f8b8eb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:36:30.289397 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c92b9ac5-a13e-4854-8748-99dd39f8b8eb admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:36:30.310957 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c008159e-98e6-4878-addb-bfeb5cf089ca admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:36:30.317441 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1146/2289] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:36:30 2022] GET /identity/v3/roles => generated 1420 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:36:30.350754 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c008159e-98e6-4878-addb-bfeb5cf089ca admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:36:30.373225 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f997d8f-5647-46a8-aaf2-e6634c43737e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:36:30.376199 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f997d8f-5647-46a8-aaf2-e6634c43737e admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:36:30.377066 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f997d8f-5647-46a8-aaf2-e6634c43737e admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:36:30.377579 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f997d8f-5647-46a8-aaf2-e6634c43737e admin admin] PATH_INFO: `/v3/projects/0c147e575342488f850408670d8838c2/users/8fc162f0d039495da2367c5189f91028/roles/a6e562cdc96c4be691d638990f8403fd` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:36:30.380305 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f997d8f-5647-46a8-aaf2-e6634c43737e admin admin] RBAC: Authorizing `identity:create_grant(project_id=0c147e575342488f850408670d8838c2, user_id=8fc162f0d039495da2367c5189f91028, role_id=a6e562cdc96c4be691d638990f8403fd)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:36:30.383564 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f997d8f-5647-46a8-aaf2-e6634c43737e admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:36:30.418116 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1144/2290] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:36:30 2022] PUT /identity/v3/projects/0c147e575342488f850408670d8838c2/users/8fc162f0d039495da2367c5189f91028/roles/a6e562cdc96c4be691d638990f8403fd => generated 0 bytes in 98 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:36:30.748752 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48584f9d-8aec-4648-ac10-44d48c37c384 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:36:30.749379 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48584f9d-8aec-4648-ac10-44d48c37c384 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:36:30.749643 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48584f9d-8aec-4648-ac10-44d48c37c384 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:36:30.838096 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-48584f9d-8aec-4648-ac10-44d48c37c384 None None] MFA Rules not processed for user `5c64b71b38a24a9d8242e8cae089410a`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:36:30.851588 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-48584f9d-8aec-4648-ac10-44d48c37c384 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:36:30.920671 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:36:31.014253 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1147/2291] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:36:30 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 268 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:36:31.032099 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f997d8f-5647-46a8-aaf2-e6634c43737e admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:36:31.039953 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-9f997d8f-5647-46a8-aaf2-e6634c43737e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:36:31.205314 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f997d8f-5647-46a8-aaf2-e6634c43737e admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:36:31.213932 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea549040-dbff-444f-9a73-e6795ac4b8da service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a35b5b7dea7b421da056f063d2436ffb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:36:31.214676 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea549040-dbff-444f-9a73-e6795ac4b8da service nova] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:36:31.216146 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea549040-dbff-444f-9a73-e6795ac4b8da service nova] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:36:31.216521 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea549040-dbff-444f-9a73-e6795ac4b8da service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:36:31.380278 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea549040-dbff-444f-9a73-e6795ac4b8da service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:36:31.387669 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea549040-dbff-444f-9a73-e6795ac4b8da service nova] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:36:31.411780 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1145/2292] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:36:31 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 382 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:36:34.135072 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a71e74f-74ce-41d9-9fd0-f6ed232c6172 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:36:34.135706 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a71e74f-74ce-41d9-9fd0-f6ed232c6172 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:36:34.135989 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a71e74f-74ce-41d9-9fd0-f6ed232c6172 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:36:34.183974 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-5a71e74f-74ce-41d9-9fd0-f6ed232c6172 None None] MFA Rules not processed for user `3376818708e443928951fd5f517c9d83`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:36:34.186901 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-5a71e74f-74ce-41d9-9fd0-f6ed232c6172 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:36:34.200156 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1148/2293] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:36:34 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 67 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 21:36:35.144526 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-269775f9-7581-4859-a550-5797ff200879 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:36:35.144994 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-269775f9-7581-4859-a550-5797ff200879 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:36:35.145484 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-269775f9-7581-4859-a550-5797ff200879 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:36:35.147316 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1146/2294] 10.222.0.90 () {54 vars in 961 bytes} [Mon Aug 29 21:36:35 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:36:35.152654 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d5da2f5-5585-4735-94a4-3fae704ca573 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:36:35.154936 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d5da2f5-5585-4735-94a4-3fae704ca573 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:36:35.155700 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d5da2f5-5585-4735-94a4-3fae704ca573 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:36:35.180573 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-8d5da2f5-5585-4735-94a4-3fae704ca573 None None] MFA Rules not processed for user `b95fa07a7c0549dbaac49406e2639e21`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:36:35.182086 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-8d5da2f5-5585-4735-94a4-3fae704ca573 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:36:35.329550 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1149/2295] 10.222.0.90 () {58 vars in 1058 bytes} [Mon Aug 29 21:36:35 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:36:35.339447 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-269775f9-7581-4859-a550-5797ff200879 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:36:35.363867 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-269775f9-7581-4859-a550-5797ff200879 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:36:35.380365 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db6d82f0-2d96-463c-81c3-5f918d314d11 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b95fa07a7c0549dbaac49406e2639e21', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:36:35.381406 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db6d82f0-2d96-463c-81c3-5f918d314d11 service placement] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:36:35.381918 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db6d82f0-2d96-463c-81c3-5f918d314d11 service placement] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:36:35.382387 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db6d82f0-2d96-463c-81c3-5f918d314d11 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:36:35.384521 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-db6d82f0-2d96-463c-81c3-5f918d314d11 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:36:35.416507 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db6d82f0-2d96-463c-81c3-5f918d314d11 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:36:35.427153 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db6d82f0-2d96-463c-81c3-5f918d314d11 service placement] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:36:35.443191 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1147/2296] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:36:35 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:36:39.416983 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb2793bb-dcd5-4bcd-b302-5fab7cd48cfe None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:36:39.417910 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb2793bb-dcd5-4bcd-b302-5fab7cd48cfe None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:36:39.418189 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb2793bb-dcd5-4bcd-b302-5fab7cd48cfe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:36:39.460794 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-cb2793bb-dcd5-4bcd-b302-5fab7cd48cfe None None] MFA Rules not processed for user `8fc162f0d039495da2367c5189f91028`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:36:39.462098 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-cb2793bb-dcd5-4bcd-b302-5fab7cd48cfe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:36:39.586171 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1150/2297] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:36:39 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 21:36:39.619457 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db6d82f0-2d96-463c-81c3-5f918d314d11 service placement] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:36:39.621611 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-db6d82f0-2d96-463c-81c3-5f918d314d11 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:36:39.811875 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db6d82f0-2d96-463c-81c3-5f918d314d11 service placement] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:36:39.817410 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5649aea0-48e2-4cae-a05d-8c7ef79d2cb0 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c63a4eaf48ad43529f0aa328f2ebfd29', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:36:39.818090 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5649aea0-48e2-4cae-a05d-8c7ef79d2cb0 service glance] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:36:39.818385 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5649aea0-48e2-4cae-a05d-8c7ef79d2cb0 service glance] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:36:39.818634 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5649aea0-48e2-4cae-a05d-8c7ef79d2cb0 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:36:39.865369 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:36:39.937858 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5649aea0-48e2-4cae-a05d-8c7ef79d2cb0 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:36:39.939355 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5649aea0-48e2-4cae-a05d-8c7ef79d2cb0 service glance] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:36:39.950761 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1148/2298] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:36:39 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 331 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:36:40.166101 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90dac7aa-036e-4cde-813b-09f63e56bdc3 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:36:40.166394 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90dac7aa-036e-4cde-813b-09f63e56bdc3 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:36:40.166675 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90dac7aa-036e-4cde-813b-09f63e56bdc3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:36:40.200935 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-90dac7aa-036e-4cde-813b-09f63e56bdc3 None None] MFA Rules not processed for user `0c0b0da87b124de1be00faec4bbdb404`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:36:40.202231 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-90dac7aa-036e-4cde-813b-09f63e56bdc3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:36:40.263405 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:36:40.294121 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1151/2299] 10.222.0.90 () {58 vars in 1009 bytes} [Mon Aug 29 21:36:40 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:36:40.305931 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5649aea0-48e2-4cae-a05d-8c7ef79d2cb0 service glance] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:36:40.307433 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-5649aea0-48e2-4cae-a05d-8c7ef79d2cb0 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:36:40.483641 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5649aea0-48e2-4cae-a05d-8c7ef79d2cb0 service glance] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:36:40.493995 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8379916b-81a9-4752-bbc7-61841304d19f service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '139bbbde6d884b42b5440727926e2c58', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:36:40.494630 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8379916b-81a9-4752-bbc7-61841304d19f service swift] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:36:40.494907 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8379916b-81a9-4752-bbc7-61841304d19f service swift] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:36:40.495171 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8379916b-81a9-4752-bbc7-61841304d19f service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:36:40.505815 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8379916b-81a9-4752-bbc7-61841304d19f service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:36:40.507591 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8379916b-81a9-4752-bbc7-61841304d19f service swift] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:36:40.519039 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1149/2300] 10.222.0.90 () {60 vars in 1384 bytes} [Mon Aug 29 21:36:40 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 213 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 29 21:36:50.286113 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=7 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:36:50.514111 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=50 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:36:51.225439 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53003505-60dd-4440-80ad-6b48f1dc90b4 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:36:51.225841 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53003505-60dd-4440-80ad-6b48f1dc90b4 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:36:51.226097 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53003505-60dd-4440-80ad-6b48f1dc90b4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:36:51.252064 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-53003505-60dd-4440-80ad-6b48f1dc90b4 None None] MFA Rules not processed for user `fdba2832a619433eb7e607bb8bf45c59`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:36:51.253254 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-53003505-60dd-4440-80ad-6b48f1dc90b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:36:51.260871 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1152/2301] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:36:51 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:36:57.254109 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccaa965b-ffe2-4c49-a865-a792de54c717 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:36:57.254648 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccaa965b-ffe2-4c49-a865-a792de54c717 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:36:57.254648 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccaa965b-ffe2-4c49-a865-a792de54c717 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:36:57.275559 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-ccaa965b-ffe2-4c49-a865-a792de54c717 None None] MFA Rules not processed for user `0c0b0da87b124de1be00faec4bbdb404`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:36:57.276851 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-ccaa965b-ffe2-4c49-a865-a792de54c717 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:36:57.290434 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1150/2302] 10.222.0.90 () {58 vars in 1009 bytes} [Mon Aug 29 21:36:57 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:36:57.305554 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53003505-60dd-4440-80ad-6b48f1dc90b4 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:36:57.324706 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53003505-60dd-4440-80ad-6b48f1dc90b4 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:36:57.333093 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c9628a1-20e9-48b3-847d-3a8b6ac5cee7 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '139bbbde6d884b42b5440727926e2c58', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:36:57.334136 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c9628a1-20e9-48b3-847d-3a8b6ac5cee7 service swift] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:36:57.334849 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c9628a1-20e9-48b3-847d-3a8b6ac5cee7 service swift] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:36:57.335359 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c9628a1-20e9-48b3-847d-3a8b6ac5cee7 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:36:57.352820 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c9628a1-20e9-48b3-847d-3a8b6ac5cee7 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:36:57.354874 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c9628a1-20e9-48b3-847d-3a8b6ac5cee7 service swift] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:36:57.365458 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1153/2303] 10.222.0.90 () {60 vars in 1384 bytes} [Mon Aug 29 21:36:57 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 60 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Aug 29 21:37:07.268881 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=3 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:37:07.362029 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=9 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:37:20.562616 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f61b3bbf-4175-43d8-8a5d-35a4285c775b None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:37:20.563210 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f61b3bbf-4175-43d8-8a5d-35a4285c775b None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:37:20.563324 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f61b3bbf-4175-43d8-8a5d-35a4285c775b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:37:20.587338 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-f61b3bbf-4175-43d8-8a5d-35a4285c775b None None] MFA Rules not processed for user `0c0b0da87b124de1be00faec4bbdb404`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:37:20.588879 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-f61b3bbf-4175-43d8-8a5d-35a4285c775b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:37:20.597423 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1151/2304] 10.222.0.90 () {58 vars in 1009 bytes} [Mon Aug 29 21:37:20 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:37:20.611036 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c9628a1-20e9-48b3-847d-3a8b6ac5cee7 service swift] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:37:20.624855 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c9628a1-20e9-48b3-847d-3a8b6ac5cee7 service swift] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:37:20.646386 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec12391e-f0f3-459b-8ef4-efbadcbdd309 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '139bbbde6d884b42b5440727926e2c58', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:37:20.647252 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec12391e-f0f3-459b-8ef4-efbadcbdd309 service swift] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:37:20.647609 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec12391e-f0f3-459b-8ef4-efbadcbdd309 service swift] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:37:20.648123 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec12391e-f0f3-459b-8ef4-efbadcbdd309 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:37:20.659826 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec12391e-f0f3-459b-8ef4-efbadcbdd309 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:37:20.661634 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec12391e-f0f3-459b-8ef4-efbadcbdd309 service swift] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:37:20.673761 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1154/2305] 10.222.0.90 () {60 vars in 1384 bytes} [Mon Aug 29 21:37:20 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 63 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 29 21:37:26.424840 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-442a218a-b54d-409e-b9f8-c6178cc26ff8 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:37:26.426035 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-442a218a-b54d-409e-b9f8-c6178cc26ff8 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:37:26.426471 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-442a218a-b54d-409e-b9f8-c6178cc26ff8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:37:26.465592 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-442a218a-b54d-409e-b9f8-c6178cc26ff8 None None] MFA Rules not processed for user `736ad47c636d4172b1a646437ac4feec`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:37:26.471470 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-442a218a-b54d-409e-b9f8-c6178cc26ff8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:37:26.603101 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1152/2306] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:37:26 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:37:30.669054 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=6 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:37:32.637506 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec12391e-f0f3-459b-8ef4-efbadcbdd309 service swift] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:37:32.643831 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-ec12391e-f0f3-459b-8ef4-efbadcbdd309 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:37:32.777051 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec12391e-f0f3-459b-8ef4-efbadcbdd309 service swift] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:37:32.782580 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfe74ee5-4864-42ce-a8b8-87065c5d4555 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:37:32.783612 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfe74ee5-4864-42ce-a8b8-87065c5d4555 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:37:32.784138 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfe74ee5-4864-42ce-a8b8-87065c5d4555 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:37:32.784806 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfe74ee5-4864-42ce-a8b8-87065c5d4555 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:37:32.801599 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfe74ee5-4864-42ce-a8b8-87065c5d4555 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:37:32.803814 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfe74ee5-4864-42ce-a8b8-87065c5d4555 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:37:32.822388 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1155/2307] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:37:32 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 186 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:37:34.707874 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-442a218a-b54d-409e-b9f8-c6178cc26ff8 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:37:34.725004 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-442a218a-b54d-409e-b9f8-c6178cc26ff8 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:37:34.732793 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e83e04f3-67c7-4971-bcca-00f2b800828e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:37:34.733735 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e83e04f3-67c7-4971-bcca-00f2b800828e service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:37:34.734201 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e83e04f3-67c7-4971-bcca-00f2b800828e service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:37:34.734667 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e83e04f3-67c7-4971-bcca-00f2b800828e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:37:34.753377 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e83e04f3-67c7-4971-bcca-00f2b800828e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:37:34.758023 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e83e04f3-67c7-4971-bcca-00f2b800828e service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:37:34.776746 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1153/2308] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:37:34 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 69 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:37:41.363434 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52c595bb-64ad-4900-9816-97d40dcdc40f None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:37:41.364046 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52c595bb-64ad-4900-9816-97d40dcdc40f None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:37:41.364046 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52c595bb-64ad-4900-9816-97d40dcdc40f None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:37:41.365364 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1156/2309] 10.222.0.90 () {54 vars in 921 bytes} [Mon Aug 29 21:37:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:37:41.374002 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e44422e6-6cee-40b8-b43e-366220f55ad4 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:37:41.374559 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e44422e6-6cee-40b8-b43e-366220f55ad4 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:37:41.374559 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e44422e6-6cee-40b8-b43e-366220f55ad4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:37:41.396462 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-e44422e6-6cee-40b8-b43e-366220f55ad4 None None] MFA Rules not processed for user `a35b5b7dea7b421da056f063d2436ffb`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:37:41.397876 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-e44422e6-6cee-40b8-b43e-366220f55ad4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:37:41.407574 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1154/2310] 10.222.0.90 () {58 vars in 1018 bytes} [Mon Aug 29 21:37:41 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:37:41.425521 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52c595bb-64ad-4900-9816-97d40dcdc40f None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:37:41.442925 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52c595bb-64ad-4900-9816-97d40dcdc40f None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:37:41.450830 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-538cc7b7-57cf-4103-82c6-79958ef72b0b service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c63a4eaf48ad43529f0aa328f2ebfd29', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:37:41.451463 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-538cc7b7-57cf-4103-82c6-79958ef72b0b service glance] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:37:41.451821 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-538cc7b7-57cf-4103-82c6-79958ef72b0b service glance] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:37:41.452092 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-538cc7b7-57cf-4103-82c6-79958ef72b0b service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:37:41.463990 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-538cc7b7-57cf-4103-82c6-79958ef72b0b service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:37:41.466078 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-538cc7b7-57cf-4103-82c6-79958ef72b0b service glance] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:37:41.477669 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1157/2311] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:37:41 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 53 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:37:41.561807 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d1d3a6-9156-4947-9073-1c2319b35999 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:37:41.562189 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d1d3a6-9156-4947-9073-1c2319b35999 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:37:41.562387 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d1d3a6-9156-4947-9073-1c2319b35999 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:37:41.563879 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1155/2312] 10.222.0.90 () {54 vars in 921 bytes} [Mon Aug 29 21:37:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:37:41.568834 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-972ebd68-09e8-48b6-8ddb-a41e4d8d9b9a None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:37:41.569281 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-972ebd68-09e8-48b6-8ddb-a41e4d8d9b9a None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:37:41.569655 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-972ebd68-09e8-48b6-8ddb-a41e4d8d9b9a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:37:41.592912 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-972ebd68-09e8-48b6-8ddb-a41e4d8d9b9a None None] MFA Rules not processed for user `0e69b7f0c3574abc9de5fb580d22152a`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:37:41.594175 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-972ebd68-09e8-48b6-8ddb-a41e4d8d9b9a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:37:41.602093 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1158/2313] 10.222.0.90 () {58 vars in 1018 bytes} [Mon Aug 29 21:37:41 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:37:41.614803 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05d1d3a6-9156-4947-9073-1c2319b35999 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:37:41.625715 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05d1d3a6-9156-4947-9073-1c2319b35999 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:37:41.629568 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b89c0db2-601a-43e1-9fff-88a63b91976a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:37:41.630319 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b89c0db2-601a-43e1-9fff-88a63b91976a service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:37:41.630515 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b89c0db2-601a-43e1-9fff-88a63b91976a service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:37:41.630800 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b89c0db2-601a-43e1-9fff-88a63b91976a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:37:41.638434 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b89c0db2-601a-43e1-9fff-88a63b91976a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:37:41.639879 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b89c0db2-601a-43e1-9fff-88a63b91976a service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:37:41.648134 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1156/2314] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:37:41 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 34 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:37:41.744180 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5855605-9355-41dd-b64b-94c37935f665 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:37:41.744474 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5855605-9355-41dd-b64b-94c37935f665 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:37:41.744753 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5855605-9355-41dd-b64b-94c37935f665 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:37:41.746828 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1159/2315] 10.222.0.90 () {54 vars in 921 bytes} [Mon Aug 29 21:37:41 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:37:41.754290 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-535a67cd-7020-4ab1-aa69-f1fb4cc7c3c0 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:37:41.754580 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-535a67cd-7020-4ab1-aa69-f1fb4cc7c3c0 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:37:41.754859 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-535a67cd-7020-4ab1-aa69-f1fb4cc7c3c0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:37:41.787952 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-535a67cd-7020-4ab1-aa69-f1fb4cc7c3c0 None None] MFA Rules not processed for user `a35b5b7dea7b421da056f063d2436ffb`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:37:41.789554 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-535a67cd-7020-4ab1-aa69-f1fb4cc7c3c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:37:41.801073 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1157/2316] 10.222.0.90 () {58 vars in 1018 bytes} [Mon Aug 29 21:37:41 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 48 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 21:37:41.814309 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5855605-9355-41dd-b64b-94c37935f665 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:37:41.826400 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5855605-9355-41dd-b64b-94c37935f665 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:37:41.831064 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a671000-b0f7-4bd8-ae95-ca4e2d3bc3a3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:37:41.831791 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a671000-b0f7-4bd8-ae95-ca4e2d3bc3a3 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:37:41.832171 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a671000-b0f7-4bd8-ae95-ca4e2d3bc3a3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:37:41.832673 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a671000-b0f7-4bd8-ae95-ca4e2d3bc3a3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:37:41.850591 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a671000-b0f7-4bd8-ae95-ca4e2d3bc3a3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:37:41.852259 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a671000-b0f7-4bd8-ae95-ca4e2d3bc3a3 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:37:41.871002 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1160/2317] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:37:41 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 57 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 21:37:51.780787 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=46 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:37:51.864526 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=41 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:37:53.237173 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a78d6f68-4426-4109-aee7-ec559ac3e1df None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:37:53.242001 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a78d6f68-4426-4109-aee7-ec559ac3e1df None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:37:53.242001 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a78d6f68-4426-4109-aee7-ec559ac3e1df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:37:53.271118 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-a78d6f68-4426-4109-aee7-ec559ac3e1df None None] MFA Rules not processed for user `0c0b0da87b124de1be00faec4bbdb404`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:37:53.272501 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-a78d6f68-4426-4109-aee7-ec559ac3e1df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:37:53.301465 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1158/2318] 10.222.0.90 () {58 vars in 1009 bytes} [Mon Aug 29 21:37:53 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 65 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:37:53.330622 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a671000-b0f7-4bd8-ae95-ca4e2d3bc3a3 service neutron] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:37:53.351032 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a671000-b0f7-4bd8-ae95-ca4e2d3bc3a3 service neutron] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:37:53.358745 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4428f37-c339-456b-99b5-e7ce1cdfbf1a service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '139bbbde6d884b42b5440727926e2c58', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:37:53.359917 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4428f37-c339-456b-99b5-e7ce1cdfbf1a service swift] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:37:53.359917 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4428f37-c339-456b-99b5-e7ce1cdfbf1a service swift] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:37:53.359917 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4428f37-c339-456b-99b5-e7ce1cdfbf1a service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:37:53.371497 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4428f37-c339-456b-99b5-e7ce1cdfbf1a service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:37:53.374509 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4428f37-c339-456b-99b5-e7ce1cdfbf1a service swift] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:37:53.396975 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1161/2319] 10.222.0.90 () {60 vars in 1384 bytes} [Mon Aug 29 21:37:53 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 68 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 29 21:38:03.262256 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=3 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:38:03.382569 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=6 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:38:09.456239 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90cec443-2641-4670-85a6-ea603907354e None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:38:09.456792 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90cec443-2641-4670-85a6-ea603907354e None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:38:09.456792 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90cec443-2641-4670-85a6-ea603907354e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:38:09.476679 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-90cec443-2641-4670-85a6-ea603907354e None None] MFA Rules not processed for user `bbbfb39899b04668ac1b7ff28d18e839`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:38:09.477929 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-90cec443-2641-4670-85a6-ea603907354e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:38:09.537982 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1159/2320] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:38:09 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:38:19.531074 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=25 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:38:21.298093 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21973746-67b0-4443-862e-3dba8e71a603 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:38:21.298982 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21973746-67b0-4443-862e-3dba8e71a603 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:38:21.298982 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21973746-67b0-4443-862e-3dba8e71a603 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:38:21.319172 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-21973746-67b0-4443-862e-3dba8e71a603 None None] MFA Rules not processed for user `0c0b0da87b124de1be00faec4bbdb404`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:38:21.320410 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-21973746-67b0-4443-862e-3dba8e71a603 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:38:21.331428 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1162/2321] 10.222.0.90 () {58 vars in 967 bytes} [Mon Aug 29 21:38:21 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:38:21.348521 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90cec443-2641-4670-85a6-ea603907354e None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:38:21.369259 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90cec443-2641-4670-85a6-ea603907354e None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:38:21.381692 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ee15cfe-11e2-412a-897a-7a863f708662 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '139bbbde6d884b42b5440727926e2c58', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:38:21.382355 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ee15cfe-11e2-412a-897a-7a863f708662 service swift] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:38:21.382698 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ee15cfe-11e2-412a-897a-7a863f708662 service swift] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:38:21.382971 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ee15cfe-11e2-412a-897a-7a863f708662 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:38:21.397485 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ee15cfe-11e2-412a-897a-7a863f708662 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:38:21.400972 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ee15cfe-11e2-412a-897a-7a863f708662 service swift] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:38:21.413509 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1160/2322] 10.222.0.90 () {60 vars in 1384 bytes} [Mon Aug 29 21:38:21 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 66 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 29 21:38:21.889593 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-017294fd-debc-437f-ad84-e58b7809bc4d None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:38:21.890270 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-017294fd-debc-437f-ad84-e58b7809bc4d None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:38:21.890547 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-017294fd-debc-437f-ad84-e58b7809bc4d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:38:21.909731 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-017294fd-debc-437f-ad84-e58b7809bc4d None None] MFA Rules not processed for user `39f161e61a514d94b6008bd006cda5ce`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:38:21.910955 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-017294fd-debc-437f-ad84-e58b7809bc4d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:38:21.978003 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1163/2323] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:38:21 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:38:31.409769 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=6 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:38:31.964841 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=28 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:38:34.857382 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ee15cfe-11e2-412a-897a-7a863f708662 service swift] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:38:34.870669 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ee15cfe-11e2-412a-897a-7a863f708662 service swift] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:38:34.877586 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1395011b-50d3-4901-a419-605173e4de67 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:38:34.878983 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1395011b-50d3-4901-a419-605173e4de67 service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:38:34.879309 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1395011b-50d3-4901-a419-605173e4de67 service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:38:34.879754 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1395011b-50d3-4901-a419-605173e4de67 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:38:34.882464 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-1395011b-50d3-4901-a419-605173e4de67 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:38:34.908492 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1395011b-50d3-4901-a419-605173e4de67 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:38:34.910762 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1395011b-50d3-4901-a419-605173e4de67 service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:38:34.925667 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1161/2324] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:38:34 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 69 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 21:38:37.413131 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-017294fd-debc-437f-ad84-e58b7809bc4d None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:38:37.424797 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-017294fd-debc-437f-ad84-e58b7809bc4d None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:38:37.429096 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a9df67d-082c-43be-b39d-ef931b7c3721 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:38:37.430480 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a9df67d-082c-43be-b39d-ef931b7c3721 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:38:37.430832 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a9df67d-082c-43be-b39d-ef931b7c3721 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:38:37.431202 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a9df67d-082c-43be-b39d-ef931b7c3721 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:38:37.433185 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-9a9df67d-082c-43be-b39d-ef931b7c3721 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:38:37.449506 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a9df67d-082c-43be-b39d-ef931b7c3721 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:38:37.451000 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a9df67d-082c-43be-b39d-ef931b7c3721 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:38:37.462743 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1164/2325] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:38:37 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 50 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:38:44.918078 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=6 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:38:47.457479 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=6 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:38:54.022961 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cb53a1d-16ac-43c2-9129-de5c1e83ee50 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:38:54.023811 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cb53a1d-16ac-43c2-9129-de5c1e83ee50 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:38:54.024626 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cb53a1d-16ac-43c2-9129-de5c1e83ee50 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:38:54.044836 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-4cb53a1d-16ac-43c2-9129-de5c1e83ee50 None None] MFA Rules not processed for user `b33a740f1c8a4d9db8d199eff66b76b2`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:38:54.046099 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-4cb53a1d-16ac-43c2-9129-de5c1e83ee50 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:38:54.107391 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1162/2326] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:38:54 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:38:54.120078 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a9df67d-082c-43be-b39d-ef931b7c3721 service neutron] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:38:54.133216 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a9df67d-082c-43be-b39d-ef931b7c3721 service neutron] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:38:54.139499 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-310d46bf-f0ce-4203-b5b0-a4cab1034b60 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a35b5b7dea7b421da056f063d2436ffb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:38:54.140129 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-310d46bf-f0ce-4203-b5b0-a4cab1034b60 service nova] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:38:54.140410 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-310d46bf-f0ce-4203-b5b0-a4cab1034b60 service nova] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:38:54.140683 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-310d46bf-f0ce-4203-b5b0-a4cab1034b60 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:38:54.142508 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-310d46bf-f0ce-4203-b5b0-a4cab1034b60 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:38:54.153809 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-310d46bf-f0ce-4203-b5b0-a4cab1034b60 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:38:54.155600 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-310d46bf-f0ce-4203-b5b0-a4cab1034b60 service nova] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:38:54.166660 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1165/2327] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:38:54 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 47 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 21:38:54.520524 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cb53a1d-16ac-43c2-9129-de5c1e83ee50 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:38:54.532747 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cb53a1d-16ac-43c2-9129-de5c1e83ee50 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:38:54.536824 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f765e2e7-88bf-4aa5-a2c6-020e3e3cf522 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:38:54.537556 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f765e2e7-88bf-4aa5-a2c6-020e3e3cf522 service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:38:54.537931 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f765e2e7-88bf-4aa5-a2c6-020e3e3cf522 service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:38:54.538250 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f765e2e7-88bf-4aa5-a2c6-020e3e3cf522 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:38:54.541262 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-f765e2e7-88bf-4aa5-a2c6-020e3e3cf522 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:38:54.553625 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f765e2e7-88bf-4aa5-a2c6-020e3e3cf522 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:38:54.555765 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f765e2e7-88bf-4aa5-a2c6-020e3e3cf522 service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:38:54.568887 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1163/2328] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:38:54 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 49 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:38:54.934129 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-310d46bf-f0ce-4203-b5b0-a4cab1034b60 service nova] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:38:54.941610 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-310d46bf-f0ce-4203-b5b0-a4cab1034b60 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:38:54.972658 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-310d46bf-f0ce-4203-b5b0-a4cab1034b60 service nova] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:38:54.985968 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d192c2a-2376-413d-bfa8-fafd0994550c service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b95fa07a7c0549dbaac49406e2639e21', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:38:54.985968 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d192c2a-2376-413d-bfa8-fafd0994550c service placement] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:38:54.985968 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d192c2a-2376-413d-bfa8-fafd0994550c service placement] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:38:54.985968 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d192c2a-2376-413d-bfa8-fafd0994550c service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:38:54.986332 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-7d192c2a-2376-413d-bfa8-fafd0994550c service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:38:54.996548 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d192c2a-2376-413d-bfa8-fafd0994550c service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:38:54.998202 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d192c2a-2376-413d-bfa8-fafd0994550c service placement] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:38:55.010523 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1166/2329] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:38:54 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 77 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 21:38:57.749831 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f765e2e7-88bf-4aa5-a2c6-020e3e3cf522 service neutron] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:38:57.753114 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-f765e2e7-88bf-4aa5-a2c6-020e3e3cf522 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:38:57.834271 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f765e2e7-88bf-4aa5-a2c6-020e3e3cf522 service neutron] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:38:57.839625 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01b04a8b-60b1-4f1f-8bf6-e771ac2dd218 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:38:57.840281 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b04a8b-60b1-4f1f-8bf6-e771ac2dd218 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:38:57.840562 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b04a8b-60b1-4f1f-8bf6-e771ac2dd218 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:38:57.840863 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b04a8b-60b1-4f1f-8bf6-e771ac2dd218 admin admin] PATH_INFO: `/v3/users/736ad47c636d4172b1a646437ac4feec` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:38:57.846933 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01b04a8b-60b1-4f1f-8bf6-e771ac2dd218 admin admin] RBAC: Authorizing `identity:delete_user(user_id=736ad47c636d4172b1a646437ac4feec)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:38:57.849077 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01b04a8b-60b1-4f1f-8bf6-e771ac2dd218 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:38:57.912022 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-01b04a8b-60b1-4f1f-8bf6-e771ac2dd218 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '736ad47c636d4172b1a646437ac4feec'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:38:57.924846 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-01b04a8b-60b1-4f1f-8bf6-e771ac2dd218 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '736ad47c636d4172b1a646437ac4feec'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:38:57.940255 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-01b04a8b-60b1-4f1f-8bf6-e771ac2dd218 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '736ad47c636d4172b1a646437ac4feec'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:38:57.940901 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-01b04a8b-60b1-4f1f-8bf6-e771ac2dd218 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '736ad47c636d4172b1a646437ac4feec'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:38:57.941748 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-01b04a8b-60b1-4f1f-8bf6-e771ac2dd218 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '736ad47c636d4172b1a646437ac4feec'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:38:57.973870 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1164/2330] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:38:57 2022] DELETE /identity/v3/users/736ad47c636d4172b1a646437ac4feec => generated 0 bytes in 230 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:38:58.196119 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d192c2a-2376-413d-bfa8-fafd0994550c service placement] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:38:58.224556 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d192c2a-2376-413d-bfa8-fafd0994550c service placement] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:38:58.323424 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7ed85e4-5eed-4671-b29b-d837c49bbc4c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:38:58.324256 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7ed85e4-5eed-4671-b29b-d837c49bbc4c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:38:58.324420 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7ed85e4-5eed-4671-b29b-d837c49bbc4c admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:38:58.324688 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7ed85e4-5eed-4671-b29b-d837c49bbc4c admin admin] PATH_INFO: `/v3/projects/f3bcb7919eee471b8dd0a728fd6255a7` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:38:58.326060 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7ed85e4-5eed-4671-b29b-d837c49bbc4c admin admin] RBAC: Authorizing `identity:delete_project(project_id=f3bcb7919eee471b8dd0a728fd6255a7)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:38:58.328023 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7ed85e4-5eed-4671-b29b-d837c49bbc4c admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:38:58.396261 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-e7ed85e4-5eed-4671-b29b-d837c49bbc4c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f3bcb7919eee471b8dd0a728fd6255a7', 'request_id': 'req-e7ed85e4-5eed-4671-b29b-d837c49bbc4c'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:38:58.404440 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-e7ed85e4-5eed-4671-b29b-d837c49bbc4c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f3bcb7919eee471b8dd0a728fd6255a7', 'request_id': 'req-e7ed85e4-5eed-4671-b29b-d837c49bbc4c'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:38:58.420388 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-e7ed85e4-5eed-4671-b29b-d837c49bbc4c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f3bcb7919eee471b8dd0a728fd6255a7', 'request_id': 'req-e7ed85e4-5eed-4671-b29b-d837c49bbc4c'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:38:58.440429 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-e7ed85e4-5eed-4671-b29b-d837c49bbc4c admin admin] The token cache is being invalidate because project f3bcb7919eee471b8dd0a728fd6255a7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:38:58.440725 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-e7ed85e4-5eed-4671-b29b-d837c49bbc4c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:38:58.441404 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-e7ed85e4-5eed-4671-b29b-d837c49bbc4c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:38:58.442970 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1167/2331] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:38:58 2022] DELETE /identity/v3/projects/f3bcb7919eee471b8dd0a728fd6255a7 => generated 0 bytes in 249 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:39:07.966754 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=84,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:39:08.437334 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=63,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:39:20.094103 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01b04a8b-60b1-4f1f-8bf6-e771ac2dd218 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:39:20.096228 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-01b04a8b-60b1-4f1f-8bf6-e771ac2dd218 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:39:20.172740 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01b04a8b-60b1-4f1f-8bf6-e771ac2dd218 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:39:20.248201 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7abc697e-82a5-4eb0-a424-3df6ae911dd3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dc226f7ddcb49ecacb51ce57fdfc11b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:39:20.248837 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7abc697e-82a5-4eb0-a424-3df6ae911dd3 service cinder] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:39:20.249119 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7abc697e-82a5-4eb0-a424-3df6ae911dd3 service cinder] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:39:20.249392 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7abc697e-82a5-4eb0-a424-3df6ae911dd3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:39:20.251489 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-7abc697e-82a5-4eb0-a424-3df6ae911dd3 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:39:20.338704 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7abc697e-82a5-4eb0-a424-3df6ae911dd3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:39:20.340258 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7abc697e-82a5-4eb0-a424-3df6ae911dd3 service cinder] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:39:20.355411 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1165/2332] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:39:20 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 262 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:39:20.765376 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbe65189-b7cf-4680-a526-483bd3ae3f9a None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:39:20.766189 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbe65189-b7cf-4680-a526-483bd3ae3f9a None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:39:20.766189 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbe65189-b7cf-4680-a526-483bd3ae3f9a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:39:20.786717 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7abc697e-82a5-4eb0-a424-3df6ae911dd3 service cinder] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:39:20.788420 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-7abc697e-82a5-4eb0-a424-3df6ae911dd3 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:39:20.789311 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-cbe65189-b7cf-4680-a526-483bd3ae3f9a None None] MFA Rules not processed for user `b6a3df8d796e4ae6ac358f103903d427`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:39:20.790624 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-cbe65189-b7cf-4680-a526-483bd3ae3f9a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:39:20.876384 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1168/2333] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:39:20 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:39:20.897447 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7abc697e-82a5-4eb0-a424-3df6ae911dd3 service cinder] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:39:20.904202 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a71f8d8-1f2b-42ed-a4d9-45ec9d5a856f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a35b5b7dea7b421da056f063d2436ffb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:39:20.904860 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a71f8d8-1f2b-42ed-a4d9-45ec9d5a856f service nova] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:39:20.905141 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a71f8d8-1f2b-42ed-a4d9-45ec9d5a856f service nova] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:39:20.905418 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a71f8d8-1f2b-42ed-a4d9-45ec9d5a856f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:39:20.907922 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-2a71f8d8-1f2b-42ed-a4d9-45ec9d5a856f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:39:20.916041 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:39:20.984783 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a71f8d8-1f2b-42ed-a4d9-45ec9d5a856f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:39:20.986733 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a71f8d8-1f2b-42ed-a4d9-45ec9d5a856f service nova] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:39:20.997492 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1166/2334] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:39:20 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 211 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 21:39:21.322615 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbe65189-b7cf-4680-a526-483bd3ae3f9a None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:39:21.324085 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-cbe65189-b7cf-4680-a526-483bd3ae3f9a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:39:21.454333 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbe65189-b7cf-4680-a526-483bd3ae3f9a None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:39:21.458247 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37ea4b61-fedd-41f5-ae6d-d0d8909b96ab service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:39:21.459090 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37ea4b61-fedd-41f5-ae6d-d0d8909b96ab service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:39:21.459375 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37ea4b61-fedd-41f5-ae6d-d0d8909b96ab service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:39:21.459650 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37ea4b61-fedd-41f5-ae6d-d0d8909b96ab service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:39:21.461629 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-37ea4b61-fedd-41f5-ae6d-d0d8909b96ab service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:39:21.472900 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37ea4b61-fedd-41f5-ae6d-d0d8909b96ab service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:39:21.474301 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37ea4b61-fedd-41f5-ae6d-d0d8909b96ab service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:39:21.483451 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1169/2335] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:39:21 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 161 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:39:27.074360 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a71f8d8-1f2b-42ed-a4d9-45ec9d5a856f service nova] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:39:27.077115 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-2a71f8d8-1f2b-42ed-a4d9-45ec9d5a856f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:39:27.097958 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a71f8d8-1f2b-42ed-a4d9-45ec9d5a856f service nova] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:39:27.122950 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c78bd79-c581-4740-949c-58d8108d8900 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a35b5b7dea7b421da056f063d2436ffb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:39:27.126348 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c78bd79-c581-4740-949c-58d8108d8900 service nova] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:39:27.126906 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c78bd79-c581-4740-949c-58d8108d8900 service nova] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:39:27.127415 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c78bd79-c581-4740-949c-58d8108d8900 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:39:27.129749 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-2c78bd79-c581-4740-949c-58d8108d8900 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:39:27.173728 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c78bd79-c581-4740-949c-58d8108d8900 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:39:27.175587 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c78bd79-c581-4740-949c-58d8108d8900 service nova] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:39:27.190818 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1167/2336] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:39:27 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 117 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:39:27.934586 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37ea4b61-fedd-41f5-ae6d-d0d8909b96ab service neutron] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:39:27.944003 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37ea4b61-fedd-41f5-ae6d-d0d8909b96ab service neutron] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:39:27.948153 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60930f79-d1e3-47e6-b4e6-1e805548b48e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:39:27.948782 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60930f79-d1e3-47e6-b4e6-1e805548b48e service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:39:27.949068 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60930f79-d1e3-47e6-b4e6-1e805548b48e service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:39:27.949349 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60930f79-d1e3-47e6-b4e6-1e805548b48e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:39:27.951588 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-60930f79-d1e3-47e6-b4e6-1e805548b48e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:39:27.972340 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60930f79-d1e3-47e6-b4e6-1e805548b48e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:39:27.980073 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60930f79-d1e3-47e6-b4e6-1e805548b48e service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:39:28.001681 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1170/2337] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:39:27 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 68 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:39:33.192899 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c78bd79-c581-4740-949c-58d8108d8900 service nova] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:39:33.233599 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c78bd79-c581-4740-949c-58d8108d8900 service nova] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:39:33.261667 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-391febfc-89f9-4561-add0-fecc93fe2585 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a35b5b7dea7b421da056f063d2436ffb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:39:33.262527 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-391febfc-89f9-4561-add0-fecc93fe2585 service nova] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:39:33.262931 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-391febfc-89f9-4561-add0-fecc93fe2585 service nova] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:39:33.263311 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-391febfc-89f9-4561-add0-fecc93fe2585 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:39:33.266599 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-391febfc-89f9-4561-add0-fecc93fe2585 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:39:33.295134 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-391febfc-89f9-4561-add0-fecc93fe2585 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:39:33.296801 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-391febfc-89f9-4561-add0-fecc93fe2585 service nova] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:39:33.325276 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1168/2338] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:39:33 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 133 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:39:33.664721 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60930f79-d1e3-47e6-b4e6-1e805548b48e service neutron] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:39:33.685380 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60930f79-d1e3-47e6-b4e6-1e805548b48e service neutron] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:39:33.692894 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8c6e4e3-71af-4411-83cf-e29f94227060 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a35b5b7dea7b421da056f063d2436ffb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:39:33.694649 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8c6e4e3-71af-4411-83cf-e29f94227060 service nova] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:39:33.695114 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8c6e4e3-71af-4411-83cf-e29f94227060 service nova] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:39:33.695548 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8c6e4e3-71af-4411-83cf-e29f94227060 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:39:33.700324 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-f8c6e4e3-71af-4411-83cf-e29f94227060 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:39:33.719318 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8c6e4e3-71af-4411-83cf-e29f94227060 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:39:33.721623 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8c6e4e3-71af-4411-83cf-e29f94227060 service nova] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:39:33.750083 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1171/2339] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:39:33 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 21:39:43.304125 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=36 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:39:43.731626 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=63 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:39:44.882111 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eec2f1a8-7547-496a-b921-f677923d1b11 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:39:44.883110 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eec2f1a8-7547-496a-b921-f677923d1b11 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:39:44.883568 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eec2f1a8-7547-496a-b921-f677923d1b11 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:39:44.910853 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-eec2f1a8-7547-496a-b921-f677923d1b11 None None] MFA Rules not processed for user `0c0b0da87b124de1be00faec4bbdb404`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:39:44.912145 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-eec2f1a8-7547-496a-b921-f677923d1b11 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:39:45.000856 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1169/2340] 10.222.0.90 () {58 vars in 967 bytes} [Mon Aug 29 21:39:44 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:39:45.056027 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8c6e4e3-71af-4411-83cf-e29f94227060 service nova] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:39:45.057706 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-f8c6e4e3-71af-4411-83cf-e29f94227060 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:39:45.147042 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8c6e4e3-71af-4411-83cf-e29f94227060 service nova] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:39:45.155094 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ef9b2e7-38d2-40c9-b907-4b6c4702362a service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '139bbbde6d884b42b5440727926e2c58', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:39:45.155598 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ef9b2e7-38d2-40c9-b907-4b6c4702362a service swift] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:39:45.155891 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ef9b2e7-38d2-40c9-b907-4b6c4702362a service swift] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:39:45.156270 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ef9b2e7-38d2-40c9-b907-4b6c4702362a service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:39:45.165701 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ef9b2e7-38d2-40c9-b907-4b6c4702362a service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:39:45.167306 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ef9b2e7-38d2-40c9-b907-4b6c4702362a service swift] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:39:45.176985 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1172/2341] 10.222.0.90 () {60 vars in 1384 bytes} [Mon Aug 29 21:39:45 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 121 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 29 21:39:45.244725 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eec2f1a8-7547-496a-b921-f677923d1b11 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:39:45.246784 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-eec2f1a8-7547-496a-b921-f677923d1b11 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:39:45.266523 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eec2f1a8-7547-496a-b921-f677923d1b11 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:39:45.270111 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64b8b894-ce79-4b4e-aff4-65621e328748 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:39:45.270731 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64b8b894-ce79-4b4e-aff4-65621e328748 service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:39:45.271014 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64b8b894-ce79-4b4e-aff4-65621e328748 service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:39:45.271817 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64b8b894-ce79-4b4e-aff4-65621e328748 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:39:45.274254 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-64b8b894-ce79-4b4e-aff4-65621e328748 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:39:45.351579 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64b8b894-ce79-4b4e-aff4-65621e328748 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:39:45.353051 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64b8b894-ce79-4b4e-aff4-65621e328748 service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:39:45.365898 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1170/2342] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:39:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 124 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:39:50.467396 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95f07ad3-913d-47a5-9b0b-61496facdc05 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:39:50.468477 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95f07ad3-913d-47a5-9b0b-61496facdc05 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:39:50.468989 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95f07ad3-913d-47a5-9b0b-61496facdc05 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:39:50.509320 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-95f07ad3-913d-47a5-9b0b-61496facdc05 None None] MFA Rules not processed for user `bbbfb39899b04668ac1b7ff28d18e839`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:39:50.513196 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-95f07ad3-913d-47a5-9b0b-61496facdc05 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:39:50.537842 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1173/2343] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:39:50 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 71 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:39:50.663555 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64b8b894-ce79-4b4e-aff4-65621e328748 service neutron] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:39:50.665107 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-64b8b894-ce79-4b4e-aff4-65621e328748 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:39:50.691198 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64b8b894-ce79-4b4e-aff4-65621e328748 service neutron] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:39:50.707194 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53ff83c3-3537-4b96-a724-9de31149f477 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:39:50.707194 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53ff83c3-3537-4b96-a724-9de31149f477 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:39:50.707194 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53ff83c3-3537-4b96-a724-9de31149f477 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:39:50.707194 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53ff83c3-3537-4b96-a724-9de31149f477 admin admin] PATH_INFO: `/v3/users/39f161e61a514d94b6008bd006cda5ce` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:39:50.710485 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53ff83c3-3537-4b96-a724-9de31149f477 admin admin] RBAC: Authorizing `identity:delete_user(user_id=39f161e61a514d94b6008bd006cda5ce)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:39:50.712531 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53ff83c3-3537-4b96-a724-9de31149f477 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:39:50.822206 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-53ff83c3-3537-4b96-a724-9de31149f477 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '39f161e61a514d94b6008bd006cda5ce'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:39:50.832625 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-53ff83c3-3537-4b96-a724-9de31149f477 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '39f161e61a514d94b6008bd006cda5ce'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:39:50.851048 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-53ff83c3-3537-4b96-a724-9de31149f477 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '39f161e61a514d94b6008bd006cda5ce'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:39:50.852301 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-53ff83c3-3537-4b96-a724-9de31149f477 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '39f161e61a514d94b6008bd006cda5ce'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:39:50.853630 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-53ff83c3-3537-4b96-a724-9de31149f477 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '39f161e61a514d94b6008bd006cda5ce'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:39:50.929252 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1171/2344] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:39:50 2022] DELETE /identity/v3/users/39f161e61a514d94b6008bd006cda5ce => generated 0 bytes in 262 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:39:51.027527 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95f07ad3-913d-47a5-9b0b-61496facdc05 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:39:51.029735 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-95f07ad3-913d-47a5-9b0b-61496facdc05 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:39:51.196293 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95f07ad3-913d-47a5-9b0b-61496facdc05 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:39:51.305556 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52256285-de09-45e1-8b03-ff0201f9be16 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b95fa07a7c0549dbaac49406e2639e21', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:39:51.306969 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52256285-de09-45e1-8b03-ff0201f9be16 service placement] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:39:51.307685 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52256285-de09-45e1-8b03-ff0201f9be16 service placement] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:39:51.308587 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52256285-de09-45e1-8b03-ff0201f9be16 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:39:51.312191 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-52256285-de09-45e1-8b03-ff0201f9be16 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:39:51.344165 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52256285-de09-45e1-8b03-ff0201f9be16 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:39:51.348205 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52256285-de09-45e1-8b03-ff0201f9be16 service placement] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:39:51.377731 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1174/2345] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:39:51 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 351 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:39:51.519111 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53ff83c3-3537-4b96-a724-9de31149f477 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:39:51.520575 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-53ff83c3-3537-4b96-a724-9de31149f477 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:39:51.551460 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:39:51.642283 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53ff83c3-3537-4b96-a724-9de31149f477 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:39:51.652081 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ae3994d-ba09-437c-8327-c709e11169fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:39:51.653226 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae3994d-ba09-437c-8327-c709e11169fd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:39:51.653737 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae3994d-ba09-437c-8327-c709e11169fd admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:39:51.654285 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae3994d-ba09-437c-8327-c709e11169fd admin admin] PATH_INFO: `/v3/projects/fb0643eb366140d59f96d86259557915` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:39:51.656190 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ae3994d-ba09-437c-8327-c709e11169fd admin admin] RBAC: Authorizing `identity:delete_project(project_id=fb0643eb366140d59f96d86259557915)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:39:51.658191 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ae3994d-ba09-437c-8327-c709e11169fd admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:39:51.721389 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-3ae3994d-ba09-437c-8327-c709e11169fd admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fb0643eb366140d59f96d86259557915', 'request_id': 'req-3ae3994d-ba09-437c-8327-c709e11169fd'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:39:51.739158 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-3ae3994d-ba09-437c-8327-c709e11169fd admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fb0643eb366140d59f96d86259557915', 'request_id': 'req-3ae3994d-ba09-437c-8327-c709e11169fd'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:39:51.755875 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-3ae3994d-ba09-437c-8327-c709e11169fd admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fb0643eb366140d59f96d86259557915', 'request_id': 'req-3ae3994d-ba09-437c-8327-c709e11169fd'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:39:51.780784 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-3ae3994d-ba09-437c-8327-c709e11169fd admin admin] The token cache is being invalidate because project fb0643eb366140d59f96d86259557915 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81434) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:39:51.781081 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-3ae3994d-ba09-437c-8327-c709e11169fd admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:39:51.782046 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-3ae3994d-ba09-437c-8327-c709e11169fd admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:39:51.784028 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1172/2346] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:39:51 2022] DELETE /identity/v3/projects/fb0643eb366140d59f96d86259557915 => generated 0 bytes in 265 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:39:51.789842 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-354785f2-8981-490b-904f-9925866b0814 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:39:51.793008 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-354785f2-8981-490b-904f-9925866b0814 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:39:51.793458 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-354785f2-8981-490b-904f-9925866b0814 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:39:51.822965 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-354785f2-8981-490b-904f-9925866b0814 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:39:51.824974 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-354785f2-8981-490b-904f-9925866b0814 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:39:51.850395 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:39:51.919730 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1175/2347] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:39:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 21:39:51.928032 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab8ef522-0218-4e25-86c9-244c7811b349 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:39:51.928683 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab8ef522-0218-4e25-86c9-244c7811b349 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:39:51.929193 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab8ef522-0218-4e25-86c9-244c7811b349 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:39:51.957408 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-ab8ef522-0218-4e25-86c9-244c7811b349 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:39:51.958947 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-ab8ef522-0218-4e25-86c9-244c7811b349 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:39:51.968782 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1173/2348] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:39:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 43 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:39:51.973168 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-354785f2-8981-490b-904f-9925866b0814 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:39:51.984425 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-354785f2-8981-490b-904f-9925866b0814 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:39:51.990377 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e17feb52-7a2c-4996-abde-4d620a944fc0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:39:51.991337 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e17feb52-7a2c-4996-abde-4d620a944fc0 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:39:51.991637 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e17feb52-7a2c-4996-abde-4d620a944fc0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:39:51.992287 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e17feb52-7a2c-4996-abde-4d620a944fc0 admin admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:39:51.993034 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e17feb52-7a2c-4996-abde-4d620a944fc0 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:39:51.993753 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e17feb52-7a2c-4996-abde-4d620a944fc0 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:39:51.997243 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e17feb52-7a2c-4996-abde-4d620a944fc0 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:39:52.022489 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1176/2349] 10.222.0.90 () {58 vars in 1153 bytes} [Mon Aug 29 21:39:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:39:52.023988 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab8ef522-0218-4e25-86c9-244c7811b349 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:39:52.041437 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab8ef522-0218-4e25-86c9-244c7811b349 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:39:52.048225 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec961b52-5a0d-4112-9cb4-2d6209470703 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:39:52.049111 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec961b52-5a0d-4112-9cb4-2d6209470703 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:39:52.049562 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec961b52-5a0d-4112-9cb4-2d6209470703 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:39:52.049875 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec961b52-5a0d-4112-9cb4-2d6209470703 admin admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:39:52.050519 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec961b52-5a0d-4112-9cb4-2d6209470703 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:39:52.053955 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec961b52-5a0d-4112-9cb4-2d6209470703 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:39:52.092029 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1174/2350] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:39:52 2022] POST /identity/v3/projects => generated 371 bytes in 68 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:39:52.096288 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e17feb52-7a2c-4996-abde-4d620a944fc0 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:39:52.113427 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e17feb52-7a2c-4996-abde-4d620a944fc0 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:39:52.206332 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93be4861-5b2e-4f86-abd6-48bca11b335c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:39:52.207047 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93be4861-5b2e-4f86-abd6-48bca11b335c admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:39:52.207518 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93be4861-5b2e-4f86-abd6-48bca11b335c admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:39:52.207750 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93be4861-5b2e-4f86-abd6-48bca11b335c admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:39:52.209191 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93be4861-5b2e-4f86-abd6-48bca11b335c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:39:52.211834 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93be4861-5b2e-4f86-abd6-48bca11b335c admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:39:52.241576 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1177/2351] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:39:52 2022] POST /identity/v3/users => generated 286 bytes in 146 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:39:52.246095 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec961b52-5a0d-4112-9cb4-2d6209470703 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:39:52.264030 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec961b52-5a0d-4112-9cb4-2d6209470703 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:39:52.269321 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-028bc82f-9cc0-41df-8bbd-f5ed9f645af5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:39:52.270001 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-028bc82f-9cc0-41df-8bbd-f5ed9f645af5 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:39:52.270286 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-028bc82f-9cc0-41df-8bbd-f5ed9f645af5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:39:52.270562 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-028bc82f-9cc0-41df-8bbd-f5ed9f645af5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:39:52.271574 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-028bc82f-9cc0-41df-8bbd-f5ed9f645af5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:39:52.273377 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-028bc82f-9cc0-41df-8bbd-f5ed9f645af5 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:39:52.289510 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1175/2352] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:39:52 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:39:52.291008 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93be4861-5b2e-4f86-abd6-48bca11b335c admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:39:52.304705 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93be4861-5b2e-4f86-abd6-48bca11b335c admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:39:52.309907 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97fb2e24-fa2c-4c9b-815d-56cf49d0c149 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:39:52.310542 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97fb2e24-fa2c-4c9b-815d-56cf49d0c149 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:39:52.310825 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97fb2e24-fa2c-4c9b-815d-56cf49d0c149 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:39:52.311102 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97fb2e24-fa2c-4c9b-815d-56cf49d0c149 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:39:52.311761 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97fb2e24-fa2c-4c9b-815d-56cf49d0c149 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:39:52.313734 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97fb2e24-fa2c-4c9b-815d-56cf49d0c149 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:39:52.328806 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1178/2353] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:39:52 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:39:52.334452 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-028bc82f-9cc0-41df-8bbd-f5ed9f645af5 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:39:52.358147 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-028bc82f-9cc0-41df-8bbd-f5ed9f645af5 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:39:52.366205 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-366dc0c0-6dbc-44a2-8bb5-d51087111d5d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:39:52.367204 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-366dc0c0-6dbc-44a2-8bb5-d51087111d5d admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:39:52.367595 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-366dc0c0-6dbc-44a2-8bb5-d51087111d5d admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:39:52.367980 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-366dc0c0-6dbc-44a2-8bb5-d51087111d5d admin admin] PATH_INFO: `/v3/projects/eaa881dfee1348ffa4feb5679ca40cb3/users/96e18bd616de4e619134dc8ead75e491/roles/02e242d5520740d4b77a5f4008c7eb57` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:39:52.389016 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-366dc0c0-6dbc-44a2-8bb5-d51087111d5d admin admin] RBAC: Authorizing `identity:create_grant(project_id=eaa881dfee1348ffa4feb5679ca40cb3, user_id=96e18bd616de4e619134dc8ead75e491, role_id=02e242d5520740d4b77a5f4008c7eb57)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:39:52.392266 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-366dc0c0-6dbc-44a2-8bb5-d51087111d5d admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:39:52.416025 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1176/2354] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:39:52 2022] PUT /identity/v3/projects/eaa881dfee1348ffa4feb5679ca40cb3/users/96e18bd616de4e619134dc8ead75e491/roles/02e242d5520740d4b77a5f4008c7eb57 => generated 0 bytes in 83 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:39:52.430222 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f9a8d01-812e-4b4b-935b-6886d8c3467f None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:39:52.430495 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f9a8d01-812e-4b4b-935b-6886d8c3467f None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:39:52.430769 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f9a8d01-812e-4b4b-935b-6886d8c3467f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:39:52.477321 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-7f9a8d01-812e-4b4b-935b-6886d8c3467f None None] MFA Rules not processed for user `96e18bd616de4e619134dc8ead75e491`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:39:52.478885 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-7f9a8d01-812e-4b4b-935b-6886d8c3467f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:39:52.611698 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1179/2355] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:39:52 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:39:52.617016 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-366dc0c0-6dbc-44a2-8bb5-d51087111d5d admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:39:52.680968 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:39:52.708987 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-366dc0c0-6dbc-44a2-8bb5-d51087111d5d admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:39:52.716860 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f82f224e-5247-444f-8a4c-fa1c38512f45 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:39:52.717497 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f82f224e-5247-444f-8a4c-fa1c38512f45 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:39:52.717826 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f82f224e-5247-444f-8a4c-fa1c38512f45 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:39:52.718112 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f82f224e-5247-444f-8a4c-fa1c38512f45 admin admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:39:52.718759 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f82f224e-5247-444f-8a4c-fa1c38512f45 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:39:52.721354 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f82f224e-5247-444f-8a4c-fa1c38512f45 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:39:52.758714 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1177/2356] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:39:52 2022] POST /identity/v3/projects => generated 371 bytes in 140 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:39:52.760901 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f9a8d01-812e-4b4b-935b-6886d8c3467f None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:39:52.783027 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f9a8d01-812e-4b4b-935b-6886d8c3467f None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:39:52.797239 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:39:52.877301 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a69a742-9dd7-49b1-b682-c13c4767a03b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:39:52.878154 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a69a742-9dd7-49b1-b682-c13c4767a03b admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:39:52.878634 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a69a742-9dd7-49b1-b682-c13c4767a03b admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:39:52.878982 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a69a742-9dd7-49b1-b682-c13c4767a03b admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:39:52.879846 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a69a742-9dd7-49b1-b682-c13c4767a03b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:39:52.882896 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a69a742-9dd7-49b1-b682-c13c4767a03b admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:39:52.927934 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1180/2357] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:39:52 2022] POST /identity/v3/users => generated 292 bytes in 165 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:39:52.929646 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f82f224e-5247-444f-8a4c-fa1c38512f45 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:39:52.951325 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f82f224e-5247-444f-8a4c-fa1c38512f45 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:39:52.972590 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-554bc60f-8930-4066-b98e-6c992fd68e1b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:39:52.973503 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-554bc60f-8930-4066-b98e-6c992fd68e1b admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:39:52.974368 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-554bc60f-8930-4066-b98e-6c992fd68e1b admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:39:52.974875 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-554bc60f-8930-4066-b98e-6c992fd68e1b admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:39:52.975898 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-554bc60f-8930-4066-b98e-6c992fd68e1b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:39:52.980744 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-554bc60f-8930-4066-b98e-6c992fd68e1b admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:39:53.006283 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1178/2358] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:39:52 2022] GET /identity/v3/roles => generated 1420 bytes in 77 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:39:53.014075 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a69a742-9dd7-49b1-b682-c13c4767a03b admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:39:53.032049 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a69a742-9dd7-49b1-b682-c13c4767a03b admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:39:53.038647 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48c8e536-59ae-4822-8523-8c0aa3bf5539 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:39:53.039959 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48c8e536-59ae-4822-8523-8c0aa3bf5539 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:39:53.040243 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48c8e536-59ae-4822-8523-8c0aa3bf5539 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:39:53.041703 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48c8e536-59ae-4822-8523-8c0aa3bf5539 admin admin] PATH_INFO: `/v3/projects/c6b34827f8484817b85e0f94880e5e2e/users/1ee7929b0aef45e18128b6aa2ea7db96/roles/a6e562cdc96c4be691d638990f8403fd` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:39:53.067048 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48c8e536-59ae-4822-8523-8c0aa3bf5539 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c6b34827f8484817b85e0f94880e5e2e, user_id=1ee7929b0aef45e18128b6aa2ea7db96, role_id=a6e562cdc96c4be691d638990f8403fd)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:39:53.072264 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48c8e536-59ae-4822-8523-8c0aa3bf5539 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:39:53.105780 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1181/2359] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:39:53 2022] PUT /identity/v3/projects/c6b34827f8484817b85e0f94880e5e2e/users/1ee7929b0aef45e18128b6aa2ea7db96/roles/a6e562cdc96c4be691d638990f8403fd => generated 0 bytes in 95 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:39:53.109771 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-554bc60f-8930-4066-b98e-6c992fd68e1b admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:39:53.122699 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-554bc60f-8930-4066-b98e-6c992fd68e1b admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:39:53.225455 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfe24ca6-fd0b-470d-8d6e-2f622b6caf73 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:39:53.236954 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfe24ca6-fd0b-470d-8d6e-2f622b6caf73 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:39:53.237482 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfe24ca6-fd0b-470d-8d6e-2f622b6caf73 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:39:53.237939 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfe24ca6-fd0b-470d-8d6e-2f622b6caf73 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:39:53.238793 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfe24ca6-fd0b-470d-8d6e-2f622b6caf73 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:39:53.241183 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfe24ca6-fd0b-470d-8d6e-2f622b6caf73 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:39:53.266620 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1179/2360] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:39:53 2022] GET /identity/v3/roles => generated 1420 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:39:53.270334 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48c8e536-59ae-4822-8523-8c0aa3bf5539 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:39:53.303897 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48c8e536-59ae-4822-8523-8c0aa3bf5539 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:39:53.321923 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a9e98a0-f3c7-4854-88fa-fee94e9c5a35 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:39:53.323298 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a9e98a0-f3c7-4854-88fa-fee94e9c5a35 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:39:53.325014 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a9e98a0-f3c7-4854-88fa-fee94e9c5a35 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:39:53.325456 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a9e98a0-f3c7-4854-88fa-fee94e9c5a35 admin admin] PATH_INFO: `/v3/projects/c6b34827f8484817b85e0f94880e5e2e/users/1ee7929b0aef45e18128b6aa2ea7db96/roles/a6e562cdc96c4be691d638990f8403fd` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:39:53.329400 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a9e98a0-f3c7-4854-88fa-fee94e9c5a35 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c6b34827f8484817b85e0f94880e5e2e, user_id=1ee7929b0aef45e18128b6aa2ea7db96, role_id=a6e562cdc96c4be691d638990f8403fd)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:39:53.334049 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a9e98a0-f3c7-4854-88fa-fee94e9c5a35 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:39:53.351542 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1182/2361] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:39:53 2022] PUT /identity/v3/projects/c6b34827f8484817b85e0f94880e5e2e/users/1ee7929b0aef45e18128b6aa2ea7db96/roles/a6e562cdc96c4be691d638990f8403fd => generated 0 bytes in 82 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:39:53.362821 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-432b03a4-ff47-486f-a11c-09e12fd9dd54 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:39:53.363469 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-432b03a4-ff47-486f-a11c-09e12fd9dd54 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:39:53.363895 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-432b03a4-ff47-486f-a11c-09e12fd9dd54 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:39:53.399279 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-432b03a4-ff47-486f-a11c-09e12fd9dd54 None None] MFA Rules not processed for user `1ee7929b0aef45e18128b6aa2ea7db96`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:39:53.400529 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-432b03a4-ff47-486f-a11c-09e12fd9dd54 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:39:53.535181 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1180/2362] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:39:53 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:39:53.549084 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a9e98a0-f3c7-4854-88fa-fee94e9c5a35 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:39:53.551161 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-4a9e98a0-f3c7-4854-88fa-fee94e9c5a35 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:39:53.679326 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a9e98a0-f3c7-4854-88fa-fee94e9c5a35 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:39:53.684129 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b16e9094-18f9-4508-9681-2448fc9cfc0e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dc226f7ddcb49ecacb51ce57fdfc11b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:39:53.684772 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b16e9094-18f9-4508-9681-2448fc9cfc0e service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:39:53.685051 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b16e9094-18f9-4508-9681-2448fc9cfc0e service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:39:53.685373 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b16e9094-18f9-4508-9681-2448fc9cfc0e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:39:53.697024 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b16e9094-18f9-4508-9681-2448fc9cfc0e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:39:53.698965 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b16e9094-18f9-4508-9681-2448fc9cfc0e service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:39:53.722733 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1183/2363] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:39:53 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 174 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:39:53.833919 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-432b03a4-ff47-486f-a11c-09e12fd9dd54 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:39:53.855607 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-432b03a4-ff47-486f-a11c-09e12fd9dd54 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:39:53.866406 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c17f7f7d-8376-4098-b15c-d0d2e805e446 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dc226f7ddcb49ecacb51ce57fdfc11b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:39:53.867761 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c17f7f7d-8376-4098-b15c-d0d2e805e446 service cinder] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:39:53.867761 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c17f7f7d-8376-4098-b15c-d0d2e805e446 service cinder] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:39:53.867761 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c17f7f7d-8376-4098-b15c-d0d2e805e446 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:39:53.916350 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:39:53.968367 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c17f7f7d-8376-4098-b15c-d0d2e805e446 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:39:53.970223 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c17f7f7d-8376-4098-b15c-d0d2e805e446 service cinder] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:39:53.996886 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1181/2364] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:39:53 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 164 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:39:55.677964 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b16e9094-18f9-4508-9681-2448fc9cfc0e service cinder] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:39:55.680069 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-b16e9094-18f9-4508-9681-2448fc9cfc0e service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:39:55.806181 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b16e9094-18f9-4508-9681-2448fc9cfc0e service cinder] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:39:55.819395 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b6f9af3-198d-4f1c-b86b-f5e60ad1acf3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:39:55.820393 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b6f9af3-198d-4f1c-b86b-f5e60ad1acf3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:39:55.820846 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b6f9af3-198d-4f1c-b86b-f5e60ad1acf3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:39:55.821453 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b6f9af3-198d-4f1c-b86b-f5e60ad1acf3 admin admin] PATH_INFO: `/v3/users/fdba2832a619433eb7e607bb8bf45c59` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:39:55.824036 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b6f9af3-198d-4f1c-b86b-f5e60ad1acf3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=fdba2832a619433eb7e607bb8bf45c59)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:39:55.826569 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b6f9af3-198d-4f1c-b86b-f5e60ad1acf3 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:39:55.840172 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:39:55.938117 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-7b6f9af3-198d-4f1c-b86b-f5e60ad1acf3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fdba2832a619433eb7e607bb8bf45c59'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:39:55.961539 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-7b6f9af3-198d-4f1c-b86b-f5e60ad1acf3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fdba2832a619433eb7e607bb8bf45c59'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:39:55.962357 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-7b6f9af3-198d-4f1c-b86b-f5e60ad1acf3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fdba2832a619433eb7e607bb8bf45c59'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:39:55.963405 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-7b6f9af3-198d-4f1c-b86b-f5e60ad1acf3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fdba2832a619433eb7e607bb8bf45c59'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:39:55.977415 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-7b6f9af3-198d-4f1c-b86b-f5e60ad1acf3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fdba2832a619433eb7e607bb8bf45c59'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:39:55.995631 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1184/2365] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:39:55 2022] DELETE /identity/v3/users/fdba2832a619433eb7e607bb8bf45c59 => generated 0 bytes in 319 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:39:56.172586 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c17f7f7d-8376-4098-b15c-d0d2e805e446 service cinder] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:39:56.185080 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c17f7f7d-8376-4098-b15c-d0d2e805e446 service cinder] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:39:56.256888 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b18669b4-5e56-4c36-aed5-29b903279ab3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:39:56.257624 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b18669b4-5e56-4c36-aed5-29b903279ab3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:39:56.257992 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b18669b4-5e56-4c36-aed5-29b903279ab3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:39:56.258320 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b18669b4-5e56-4c36-aed5-29b903279ab3 admin admin] PATH_INFO: `/v3/projects/df74318c1e404179bbb8533b5909deb8` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:39:56.259758 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b18669b4-5e56-4c36-aed5-29b903279ab3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=df74318c1e404179bbb8533b5909deb8)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:39:56.261897 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b18669b4-5e56-4c36-aed5-29b903279ab3 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:39:56.316228 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-b18669b4-5e56-4c36-aed5-29b903279ab3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'df74318c1e404179bbb8533b5909deb8', 'request_id': 'req-b18669b4-5e56-4c36-aed5-29b903279ab3'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:39:56.339712 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-b18669b4-5e56-4c36-aed5-29b903279ab3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'df74318c1e404179bbb8533b5909deb8', 'request_id': 'req-b18669b4-5e56-4c36-aed5-29b903279ab3'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:39:56.346051 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-b18669b4-5e56-4c36-aed5-29b903279ab3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'df74318c1e404179bbb8533b5909deb8', 'request_id': 'req-b18669b4-5e56-4c36-aed5-29b903279ab3'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:39:56.360535 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-b18669b4-5e56-4c36-aed5-29b903279ab3 admin admin] The token cache is being invalidate because project df74318c1e404179bbb8533b5909deb8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81434) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:39:56.360816 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-b18669b4-5e56-4c36-aed5-29b903279ab3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:39:56.361457 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-b18669b4-5e56-4c36-aed5-29b903279ab3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:39:56.363006 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1182/2366] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:39:56 2022] DELETE /identity/v3/projects/df74318c1e404179bbb8533b5909deb8 => generated 0 bytes in 191 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:39:56.367352 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b6f9af3-198d-4f1c-b86b-f5e60ad1acf3 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:39:56.368785 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-7b6f9af3-198d-4f1c-b86b-f5e60ad1acf3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:39:56.382513 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b6f9af3-198d-4f1c-b86b-f5e60ad1acf3 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:39:56.440071 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e56f6cb1-d890-4103-bb57-f1752e5aa06c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:39:56.440631 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e56f6cb1-d890-4103-bb57-f1752e5aa06c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:39:56.440911 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e56f6cb1-d890-4103-bb57-f1752e5aa06c admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:39:56.441186 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e56f6cb1-d890-4103-bb57-f1752e5aa06c admin admin] PATH_INFO: `/v3/users/8fc162f0d039495da2367c5189f91028` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:39:56.442380 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e56f6cb1-d890-4103-bb57-f1752e5aa06c admin admin] RBAC: Authorizing `identity:delete_user(user_id=8fc162f0d039495da2367c5189f91028)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:39:56.444464 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e56f6cb1-d890-4103-bb57-f1752e5aa06c admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:39:56.498855 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-e56f6cb1-d890-4103-bb57-f1752e5aa06c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8fc162f0d039495da2367c5189f91028'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:39:56.512879 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-e56f6cb1-d890-4103-bb57-f1752e5aa06c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8fc162f0d039495da2367c5189f91028'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:39:56.513628 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-e56f6cb1-d890-4103-bb57-f1752e5aa06c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8fc162f0d039495da2367c5189f91028'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:39:56.514283 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-e56f6cb1-d890-4103-bb57-f1752e5aa06c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8fc162f0d039495da2367c5189f91028'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:39:56.524585 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-e56f6cb1-d890-4103-bb57-f1752e5aa06c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8fc162f0d039495da2367c5189f91028'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:39:56.541142 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1185/2367] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:39:56 2022] DELETE /identity/v3/users/8fc162f0d039495da2367c5189f91028 => generated 0 bytes in 174 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:39:56.744295 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b18669b4-5e56-4c36-aed5-29b903279ab3 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:39:56.745759 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-b18669b4-5e56-4c36-aed5-29b903279ab3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:39:56.766867 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b18669b4-5e56-4c36-aed5-29b903279ab3 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:39:56.846039 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72aad9bc-5e5d-4729-8482-cdc152310b80 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:39:56.846692 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72aad9bc-5e5d-4729-8482-cdc152310b80 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:39:56.846972 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72aad9bc-5e5d-4729-8482-cdc152310b80 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:39:56.847243 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72aad9bc-5e5d-4729-8482-cdc152310b80 admin admin] PATH_INFO: `/v3/projects/0c147e575342488f850408670d8838c2` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:39:56.848421 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72aad9bc-5e5d-4729-8482-cdc152310b80 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0c147e575342488f850408670d8838c2)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:39:56.850856 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72aad9bc-5e5d-4729-8482-cdc152310b80 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:39:56.862161 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:39:56.933684 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-72aad9bc-5e5d-4729-8482-cdc152310b80 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0c147e575342488f850408670d8838c2', 'request_id': 'req-72aad9bc-5e5d-4729-8482-cdc152310b80'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:39:56.945627 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-72aad9bc-5e5d-4729-8482-cdc152310b80 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0c147e575342488f850408670d8838c2', 'request_id': 'req-72aad9bc-5e5d-4729-8482-cdc152310b80'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:39:56.961192 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-72aad9bc-5e5d-4729-8482-cdc152310b80 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0c147e575342488f850408670d8838c2', 'request_id': 'req-72aad9bc-5e5d-4729-8482-cdc152310b80'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:39:56.987658 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-72aad9bc-5e5d-4729-8482-cdc152310b80 admin admin] The token cache is being invalidate because project 0c147e575342488f850408670d8838c2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81434) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:39:56.987960 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-72aad9bc-5e5d-4729-8482-cdc152310b80 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:39:56.988673 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-72aad9bc-5e5d-4729-8482-cdc152310b80 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:39:56.991517 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1183/2368] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:39:56 2022] DELETE /identity/v3/projects/0c147e575342488f850408670d8838c2 => generated 0 bytes in 248 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:40:06.536508 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=76,keystone:DELETE=20,keystone:INSERT=2 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:40:06.991914 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=23,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:40:14.864054 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e84a5766-33fa-4ced-ad58-097070594170 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:40:14.864623 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e84a5766-33fa-4ced-ad58-097070594170 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:40:14.864846 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e84a5766-33fa-4ced-ad58-097070594170 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:40:14.866540 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1186/2369] 10.222.0.90 () {54 vars in 921 bytes} [Mon Aug 29 21:40:14 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:40:14.873570 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ef63ec7-b7ab-4797-ba0f-b8062e0d68c0 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:40:14.874168 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ef63ec7-b7ab-4797-ba0f-b8062e0d68c0 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:40:14.874168 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ef63ec7-b7ab-4797-ba0f-b8062e0d68c0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:40:14.903195 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-0ef63ec7-b7ab-4797-ba0f-b8062e0d68c0 None None] MFA Rules not processed for user `0e69b7f0c3574abc9de5fb580d22152a`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:40:14.904579 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-0ef63ec7-b7ab-4797-ba0f-b8062e0d68c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:40:14.972620 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1184/2370] 10.222.0.90 () {58 vars in 1018 bytes} [Mon Aug 29 21:40:14 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 21:40:14.988928 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e84a5766-33fa-4ced-ad58-097070594170 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:40:14.991778 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-e84a5766-33fa-4ced-ad58-097070594170 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:40:15.018524 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e84a5766-33fa-4ced-ad58-097070594170 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:40:15.024461 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2ea931e-0459-43f8-9c3f-bf8300156118 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:40:15.025106 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2ea931e-0459-43f8-9c3f-bf8300156118 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:40:15.025482 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2ea931e-0459-43f8-9c3f-bf8300156118 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:40:15.025948 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2ea931e-0459-43f8-9c3f-bf8300156118 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:40:15.039981 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2ea931e-0459-43f8-9c3f-bf8300156118 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:40:15.043097 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2ea931e-0459-43f8-9c3f-bf8300156118 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:40:15.056383 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1187/2371] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:40:14 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 69 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:40:15.179290 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb67cc84-ffa3-41e2-89a9-fa846e4494f9 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:40:15.179678 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb67cc84-ffa3-41e2-89a9-fa846e4494f9 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:40:15.180034 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb67cc84-ffa3-41e2-89a9-fa846e4494f9 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:40:15.181571 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1185/2372] 10.222.0.90 () {54 vars in 921 bytes} [Mon Aug 29 21:40:15 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 21:40:15.188388 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-535fb6ce-eb74-42ef-b140-62325969dd1e None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:40:15.188727 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-535fb6ce-eb74-42ef-b140-62325969dd1e None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:40:15.188978 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-535fb6ce-eb74-42ef-b140-62325969dd1e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:40:15.212351 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-535fb6ce-eb74-42ef-b140-62325969dd1e None None] MFA Rules not processed for user `a35b5b7dea7b421da056f063d2436ffb`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:40:15.213683 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-535fb6ce-eb74-42ef-b140-62325969dd1e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:40:15.300748 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1188/2373] 10.222.0.90 () {58 vars in 1018 bytes} [Mon Aug 29 21:40:15 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 21:40:15.315954 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb67cc84-ffa3-41e2-89a9-fa846e4494f9 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:40:15.318790 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-eb67cc84-ffa3-41e2-89a9-fa846e4494f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:40:15.339669 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb67cc84-ffa3-41e2-89a9-fa846e4494f9 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:40:15.343563 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abb3024b-97c9-4843-b4b5-cfab867c5e39 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:40:15.344281 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abb3024b-97c9-4843-b4b5-cfab867c5e39 service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:40:15.344761 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abb3024b-97c9-4843-b4b5-cfab867c5e39 service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:40:15.345037 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abb3024b-97c9-4843-b4b5-cfab867c5e39 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:40:15.358109 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abb3024b-97c9-4843-b4b5-cfab867c5e39 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:40:15.359834 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abb3024b-97c9-4843-b4b5-cfab867c5e39 service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:40:15.374356 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1186/2374] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:40:15 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 59 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:40:21.185165 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-535fb6ce-eb74-42ef-b140-62325969dd1e None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:40:21.187279 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-535fb6ce-eb74-42ef-b140-62325969dd1e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:40:21.206106 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-535fb6ce-eb74-42ef-b140-62325969dd1e None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:40:21.212432 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cad0ba9-226a-4684-aaad-90c40d2dc779 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a35b5b7dea7b421da056f063d2436ffb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:40:21.213283 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cad0ba9-226a-4684-aaad-90c40d2dc779 service nova] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:40:21.213662 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cad0ba9-226a-4684-aaad-90c40d2dc779 service nova] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:40:21.214970 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cad0ba9-226a-4684-aaad-90c40d2dc779 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:40:21.216074 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-9cad0ba9-226a-4684-aaad-90c40d2dc779 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:40:21.296933 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cad0ba9-226a-4684-aaad-90c40d2dc779 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:40:21.298720 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cad0ba9-226a-4684-aaad-90c40d2dc779 service nova] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:40:21.310201 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1189/2375] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:40:21 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 126 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:40:25.367041 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=29 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:40:30.359639 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abb3024b-97c9-4843-b4b5-cfab867c5e39 service neutron] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:40:30.372751 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abb3024b-97c9-4843-b4b5-cfab867c5e39 service neutron] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:40:30.376838 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c67f4f6-24e8-422e-9947-49f21a14d729 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:40:30.377511 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c67f4f6-24e8-422e-9947-49f21a14d729 service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:40:30.377905 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c67f4f6-24e8-422e-9947-49f21a14d729 service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:40:30.379635 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c67f4f6-24e8-422e-9947-49f21a14d729 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:40:30.390698 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-0c67f4f6-24e8-422e-9947-49f21a14d729 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:40:30.480938 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c67f4f6-24e8-422e-9947-49f21a14d729 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:40:30.481820 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c67f4f6-24e8-422e-9947-49f21a14d729 service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:40:30.494031 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1187/2376] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:40:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 135 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:40:31.305092 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=65 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:40:34.942638 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cad0ba9-226a-4684-aaad-90c40d2dc779 service nova] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:40:34.944124 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-9cad0ba9-226a-4684-aaad-90c40d2dc779 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:40:34.969054 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cad0ba9-226a-4684-aaad-90c40d2dc779 service nova] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:40:34.975278 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f85cec17-f2cb-4bd1-b1f1-4f90efacd90e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:40:34.977149 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f85cec17-f2cb-4bd1-b1f1-4f90efacd90e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:40:34.977424 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f85cec17-f2cb-4bd1-b1f1-4f90efacd90e admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:40:34.977898 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f85cec17-f2cb-4bd1-b1f1-4f90efacd90e admin admin] PATH_INFO: `/v3/users/b33a740f1c8a4d9db8d199eff66b76b2` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:40:34.979284 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f85cec17-f2cb-4bd1-b1f1-4f90efacd90e admin admin] RBAC: Authorizing `identity:delete_user(user_id=b33a740f1c8a4d9db8d199eff66b76b2)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:40:34.981193 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f85cec17-f2cb-4bd1-b1f1-4f90efacd90e admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:40:35.046680 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-f85cec17-f2cb-4bd1-b1f1-4f90efacd90e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b33a740f1c8a4d9db8d199eff66b76b2'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:40:35.063983 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-f85cec17-f2cb-4bd1-b1f1-4f90efacd90e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b33a740f1c8a4d9db8d199eff66b76b2'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:40:35.064719 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-f85cec17-f2cb-4bd1-b1f1-4f90efacd90e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b33a740f1c8a4d9db8d199eff66b76b2'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:40:35.065441 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-f85cec17-f2cb-4bd1-b1f1-4f90efacd90e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b33a740f1c8a4d9db8d199eff66b76b2'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:40:35.075306 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-f85cec17-f2cb-4bd1-b1f1-4f90efacd90e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b33a740f1c8a4d9db8d199eff66b76b2'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:40:35.080059 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c67f4f6-24e8-422e-9947-49f21a14d729 service neutron] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:40:35.082971 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-0c67f4f6-24e8-422e-9947-49f21a14d729 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:40:35.097088 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c67f4f6-24e8-422e-9947-49f21a14d729 service neutron] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:40:35.099229 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1190/2377] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:40:34 2022] DELETE /identity/v3/users/b33a740f1c8a4d9db8d199eff66b76b2 => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:40:35.199439 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16ef06ac-bea2-498d-83d2-cd7eab0971d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:40:35.201154 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16ef06ac-bea2-498d-83d2-cd7eab0971d3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:40:35.201437 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16ef06ac-bea2-498d-83d2-cd7eab0971d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:40:35.201946 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16ef06ac-bea2-498d-83d2-cd7eab0971d3 admin admin] PATH_INFO: `/v3/users/96e18bd616de4e619134dc8ead75e491` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:40:35.204472 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16ef06ac-bea2-498d-83d2-cd7eab0971d3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=96e18bd616de4e619134dc8ead75e491)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:40:35.207233 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16ef06ac-bea2-498d-83d2-cd7eab0971d3 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:40:35.277206 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-16ef06ac-bea2-498d-83d2-cd7eab0971d3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '96e18bd616de4e619134dc8ead75e491'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:40:35.287398 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-16ef06ac-bea2-498d-83d2-cd7eab0971d3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '96e18bd616de4e619134dc8ead75e491'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:40:35.301049 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-16ef06ac-bea2-498d-83d2-cd7eab0971d3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '96e18bd616de4e619134dc8ead75e491'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:40:35.302844 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-16ef06ac-bea2-498d-83d2-cd7eab0971d3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '96e18bd616de4e619134dc8ead75e491'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:40:35.304200 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-16ef06ac-bea2-498d-83d2-cd7eab0971d3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '96e18bd616de4e619134dc8ead75e491'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:40:35.332836 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1188/2378] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:40:35 2022] DELETE /identity/v3/users/96e18bd616de4e619134dc8ead75e491 => generated 0 bytes in 254 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 21:40:35.334432 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f85cec17-f2cb-4bd1-b1f1-4f90efacd90e admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:40:35.335975 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-f85cec17-f2cb-4bd1-b1f1-4f90efacd90e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:40:35.337533 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c2ea080-aa09-48d8-92e3-e5f9652180a5 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:40:35.337975 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c2ea080-aa09-48d8-92e3-e5f9652180a5 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:40:35.338372 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c2ea080-aa09-48d8-92e3-e5f9652180a5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:40:35.352565 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f85cec17-f2cb-4bd1-b1f1-4f90efacd90e admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:40:35.366464 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-8c2ea080-aa09-48d8-92e3-e5f9652180a5 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:40:35.367764 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-8c2ea080-aa09-48d8-92e3-e5f9652180a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:40:35.376545 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:40:35.422961 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5457e0a-1b79-4d11-9c04-6a4e852d915b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:40:35.423601 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5457e0a-1b79-4d11-9c04-6a4e852d915b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:40:35.423901 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5457e0a-1b79-4d11-9c04-6a4e852d915b admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:40:35.424155 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5457e0a-1b79-4d11-9c04-6a4e852d915b admin admin] PATH_INFO: `/v3/projects/a2847581073e482d8ec3090bcafe93a7` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:40:35.425330 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5457e0a-1b79-4d11-9c04-6a4e852d915b admin admin] RBAC: Authorizing `identity:delete_project(project_id=a2847581073e482d8ec3090bcafe93a7)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:40:35.427560 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5457e0a-1b79-4d11-9c04-6a4e852d915b admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:40:35.443091 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1189/2379] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:40:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:40:35.454300 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c2ea080-aa09-48d8-92e3-e5f9652180a5 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:40:35.455676 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-8c2ea080-aa09-48d8-92e3-e5f9652180a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:40:35.494888 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-e5457e0a-1b79-4d11-9c04-6a4e852d915b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a2847581073e482d8ec3090bcafe93a7', 'request_id': 'req-e5457e0a-1b79-4d11-9c04-6a4e852d915b'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:40:35.509648 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-e5457e0a-1b79-4d11-9c04-6a4e852d915b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a2847581073e482d8ec3090bcafe93a7', 'request_id': 'req-e5457e0a-1b79-4d11-9c04-6a4e852d915b'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:40:35.523345 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-e5457e0a-1b79-4d11-9c04-6a4e852d915b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a2847581073e482d8ec3090bcafe93a7', 'request_id': 'req-e5457e0a-1b79-4d11-9c04-6a4e852d915b'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:40:35.538940 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-e5457e0a-1b79-4d11-9c04-6a4e852d915b admin admin] The token cache is being invalidate because project a2847581073e482d8ec3090bcafe93a7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:40:35.539290 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-e5457e0a-1b79-4d11-9c04-6a4e852d915b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:40:35.540018 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-e5457e0a-1b79-4d11-9c04-6a4e852d915b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:40:35.542085 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1191/2380] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:40:35 2022] DELETE /identity/v3/projects/a2847581073e482d8ec3090bcafe93a7 => generated 0 bytes in 208 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:40:35.546645 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1665ba57-6164-42e6-975d-c52db4b7961f None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:40:35.546935 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1665ba57-6164-42e6-975d-c52db4b7961f None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:40:35.547219 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1665ba57-6164-42e6-975d-c52db4b7961f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:40:35.549064 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c2ea080-aa09-48d8-92e3-e5f9652180a5 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:40:35.555800 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:40:35.581583 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-1665ba57-6164-42e6-975d-c52db4b7961f None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:40:35.583616 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-1665ba57-6164-42e6-975d-c52db4b7961f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:40:35.623231 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a54ddfe-0489-437a-8fd8-e0d6cc4ceb14 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:40:35.623869 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a54ddfe-0489-437a-8fd8-e0d6cc4ceb14 service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:40:35.624155 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a54ddfe-0489-437a-8fd8-e0d6cc4ceb14 service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:40:35.624430 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a54ddfe-0489-437a-8fd8-e0d6cc4ceb14 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:40:35.628009 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-2a54ddfe-0489-437a-8fd8-e0d6cc4ceb14 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:40:35.696974 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1192/2381] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:40:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 21:40:35.705050 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd6d4b48-5124-499f-9bf5-4c7f70b29d07 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:40:35.705586 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd6d4b48-5124-499f-9bf5-4c7f70b29d07 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:40:35.705975 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd6d4b48-5124-499f-9bf5-4c7f70b29d07 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:40:35.711625 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a54ddfe-0489-437a-8fd8-e0d6cc4ceb14 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:40:35.713494 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a54ddfe-0489-437a-8fd8-e0d6cc4ceb14 service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:40:35.725078 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1190/2382] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:40:35 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 272 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 21:40:35.728604 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-bd6d4b48-5124-499f-9bf5-4c7f70b29d07 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:40:35.730258 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-bd6d4b48-5124-499f-9bf5-4c7f70b29d07 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:40:35.742389 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1193/2383] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:40:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 21:40:35.746496 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a54ddfe-0489-437a-8fd8-e0d6cc4ceb14 service neutron] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:40:35.759065 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a54ddfe-0489-437a-8fd8-e0d6cc4ceb14 service neutron] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:40:35.764120 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2d0caa6-e9c1-45eb-ba81-5d4d9c67ba77 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:40:35.765110 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2d0caa6-e9c1-45eb-ba81-5d4d9c67ba77 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:40:35.765110 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2d0caa6-e9c1-45eb-ba81-5d4d9c67ba77 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:40:35.765333 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2d0caa6-e9c1-45eb-ba81-5d4d9c67ba77 admin admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:40:35.766263 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2d0caa6-e9c1-45eb-ba81-5d4d9c67ba77 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:40:35.766692 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2d0caa6-e9c1-45eb-ba81-5d4d9c67ba77 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:40:35.768535 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2d0caa6-e9c1-45eb-ba81-5d4d9c67ba77 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:40:35.783782 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1191/2384] 10.222.0.90 () {58 vars in 1153 bytes} [Mon Aug 29 21:40:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:40:35.789703 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd6d4b48-5124-499f-9bf5-4c7f70b29d07 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:40:35.804758 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd6d4b48-5124-499f-9bf5-4c7f70b29d07 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:40:35.810283 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bdc55c6-9ee9-413a-bb78-bf7f52190d70 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:40:35.811009 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bdc55c6-9ee9-413a-bb78-bf7f52190d70 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:40:35.811288 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bdc55c6-9ee9-413a-bb78-bf7f52190d70 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:40:35.811560 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bdc55c6-9ee9-413a-bb78-bf7f52190d70 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:40:35.812276 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bdc55c6-9ee9-413a-bb78-bf7f52190d70 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:40:35.814368 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bdc55c6-9ee9-413a-bb78-bf7f52190d70 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:40:35.849035 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1194/2385] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:40:35 2022] POST /identity/v3/projects => generated 373 bytes in 61 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:40:35.853444 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2d0caa6-e9c1-45eb-ba81-5d4d9c67ba77 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:40:35.873849 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2d0caa6-e9c1-45eb-ba81-5d4d9c67ba77 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:40:35.879225 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:40:35.955711 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56675544-ef84-41a8-af7c-3b129831fc38 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:40:35.956336 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56675544-ef84-41a8-af7c-3b129831fc38 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:40:35.956616 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56675544-ef84-41a8-af7c-3b129831fc38 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:40:35.957144 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56675544-ef84-41a8-af7c-3b129831fc38 admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:40:35.958412 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56675544-ef84-41a8-af7c-3b129831fc38 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:40:35.960317 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56675544-ef84-41a8-af7c-3b129831fc38 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:40:35.989921 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1192/2386] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:40:35 2022] POST /identity/v3/users => generated 287 bytes in 137 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:40:35.994182 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bdc55c6-9ee9-413a-bb78-bf7f52190d70 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:40:36.005767 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bdc55c6-9ee9-413a-bb78-bf7f52190d70 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:40:36.010883 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaa1d6d0-35df-4484-865e-6b6924cc2c65 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:40:36.011593 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaa1d6d0-35df-4484-865e-6b6924cc2c65 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:40:36.011876 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaa1d6d0-35df-4484-865e-6b6924cc2c65 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:40:36.012218 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaa1d6d0-35df-4484-865e-6b6924cc2c65 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:40:36.012931 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaa1d6d0-35df-4484-865e-6b6924cc2c65 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:40:36.014635 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaa1d6d0-35df-4484-865e-6b6924cc2c65 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:40:36.027906 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1195/2387] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:40:35 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:40:36.032307 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56675544-ef84-41a8-af7c-3b129831fc38 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:40:36.043683 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56675544-ef84-41a8-af7c-3b129831fc38 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:40:36.048355 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97dbd5f3-9e77-47c0-a877-75fab3dd9f8d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:40:36.049095 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97dbd5f3-9e77-47c0-a877-75fab3dd9f8d admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:40:36.049379 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97dbd5f3-9e77-47c0-a877-75fab3dd9f8d admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:40:36.049654 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97dbd5f3-9e77-47c0-a877-75fab3dd9f8d admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:40:36.050369 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97dbd5f3-9e77-47c0-a877-75fab3dd9f8d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:40:36.052628 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97dbd5f3-9e77-47c0-a877-75fab3dd9f8d admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:40:36.063379 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1193/2388] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:40:36 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:40:36.069080 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaa1d6d0-35df-4484-865e-6b6924cc2c65 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:40:36.083656 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaa1d6d0-35df-4484-865e-6b6924cc2c65 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:40:36.089848 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0abb74d-5f84-4656-8edd-83c70842ca68 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:40:36.090530 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0abb74d-5f84-4656-8edd-83c70842ca68 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:40:36.090809 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0abb74d-5f84-4656-8edd-83c70842ca68 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:40:36.091671 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0abb74d-5f84-4656-8edd-83c70842ca68 admin admin] PATH_INFO: `/v3/projects/1c8fe22edbdd4ab5b4006dc5c526ec5f/users/3be7a2a7ca204a5aa38907b9d08cdf7b/roles/02e242d5520740d4b77a5f4008c7eb57` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:40:36.107637 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0abb74d-5f84-4656-8edd-83c70842ca68 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1c8fe22edbdd4ab5b4006dc5c526ec5f, user_id=3be7a2a7ca204a5aa38907b9d08cdf7b, role_id=02e242d5520740d4b77a5f4008c7eb57)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:40:36.110643 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0abb74d-5f84-4656-8edd-83c70842ca68 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:40:36.126883 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1196/2389] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:40:36 2022] PUT /identity/v3/projects/1c8fe22edbdd4ab5b4006dc5c526ec5f/users/3be7a2a7ca204a5aa38907b9d08cdf7b/roles/02e242d5520740d4b77a5f4008c7eb57 => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:40:36.139304 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb6c5f3-e90c-4086-9672-b8f806499165 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:40:36.139932 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb6c5f3-e90c-4086-9672-b8f806499165 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:40:36.140235 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb6c5f3-e90c-4086-9672-b8f806499165 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:40:36.183177 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0abb74d-5f84-4656-8edd-83c70842ca68 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:40:36.185365 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-e0abb74d-5f84-4656-8edd-83c70842ca68 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:40:36.187069 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-1cb6c5f3-e90c-4086-9672-b8f806499165 None None] MFA Rules not processed for user `3be7a2a7ca204a5aa38907b9d08cdf7b`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:40:36.189005 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-1cb6c5f3-e90c-4086-9672-b8f806499165 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:40:36.261349 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0abb74d-5f84-4656-8edd-83c70842ca68 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:40:36.266041 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dc82626-1118-4dbc-ad52-70f6c38cdce3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:40:36.266666 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dc82626-1118-4dbc-ad52-70f6c38cdce3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:40:36.266945 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dc82626-1118-4dbc-ad52-70f6c38cdce3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:40:36.267217 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dc82626-1118-4dbc-ad52-70f6c38cdce3 admin admin] PATH_INFO: `/v3/projects/eaa881dfee1348ffa4feb5679ca40cb3` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:40:36.267740 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1194/2390] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:40:36 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 21:40:36.268338 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dc82626-1118-4dbc-ad52-70f6c38cdce3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=eaa881dfee1348ffa4feb5679ca40cb3)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:40:36.270313 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dc82626-1118-4dbc-ad52-70f6c38cdce3 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:40:36.278178 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cb6c5f3-e90c-4086-9672-b8f806499165 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:40:36.279587 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-1cb6c5f3-e90c-4086-9672-b8f806499165 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:40:36.327754 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=95,keystone:INSERT=2,keystone:DELETE=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:40:36.340051 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-7dc82626-1118-4dbc-ad52-70f6c38cdce3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'eaa881dfee1348ffa4feb5679ca40cb3', 'request_id': 'req-7dc82626-1118-4dbc-ad52-70f6c38cdce3'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:40:36.350442 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-7dc82626-1118-4dbc-ad52-70f6c38cdce3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'eaa881dfee1348ffa4feb5679ca40cb3', 'request_id': 'req-7dc82626-1118-4dbc-ad52-70f6c38cdce3'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:40:36.358580 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-7dc82626-1118-4dbc-ad52-70f6c38cdce3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'eaa881dfee1348ffa4feb5679ca40cb3', 'request_id': 'req-7dc82626-1118-4dbc-ad52-70f6c38cdce3'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:40:36.369025 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cb6c5f3-e90c-4086-9672-b8f806499165 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:40:36.371087 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-7dc82626-1118-4dbc-ad52-70f6c38cdce3 admin admin] The token cache is being invalidate because project eaa881dfee1348ffa4feb5679ca40cb3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:40:36.371411 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-7dc82626-1118-4dbc-ad52-70f6c38cdce3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:40:36.372057 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-7dc82626-1118-4dbc-ad52-70f6c38cdce3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:40:36.374352 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1197/2391] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:40:36 2022] DELETE /identity/v3/projects/eaa881dfee1348ffa4feb5679ca40cb3 => generated 0 bytes in 191 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:40:36.377763 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dc82626-1118-4dbc-ad52-70f6c38cdce3 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:40:36.379172 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-7dc82626-1118-4dbc-ad52-70f6c38cdce3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:40:36.380422 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:40:36.443417 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e75a443f-a83f-45c3-9180-1d07ab868d8e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a35b5b7dea7b421da056f063d2436ffb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:40:36.444062 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e75a443f-a83f-45c3-9180-1d07ab868d8e service nova] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:40:36.444377 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e75a443f-a83f-45c3-9180-1d07ab868d8e service nova] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:40:36.444712 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e75a443f-a83f-45c3-9180-1d07ab868d8e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:40:36.447237 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-e75a443f-a83f-45c3-9180-1d07ab868d8e service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:40:36.454735 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dc82626-1118-4dbc-ad52-70f6c38cdce3 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:40:36.460104 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1aa911d-4a99-495d-bcf4-e6a3388bee56 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:40:36.460762 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1aa911d-4a99-495d-bcf4-e6a3388bee56 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:40:36.461046 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1aa911d-4a99-495d-bcf4-e6a3388bee56 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:40:36.461318 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1aa911d-4a99-495d-bcf4-e6a3388bee56 admin admin] PATH_INFO: `/v3/users/1ee7929b0aef45e18128b6aa2ea7db96` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:40:36.462526 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1aa911d-4a99-495d-bcf4-e6a3388bee56 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1ee7929b0aef45e18128b6aa2ea7db96)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:40:36.464454 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1aa911d-4a99-495d-bcf4-e6a3388bee56 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:40:36.526310 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e75a443f-a83f-45c3-9180-1d07ab868d8e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:40:36.528181 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e75a443f-a83f-45c3-9180-1d07ab868d8e service nova] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:40:36.538514 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-f1aa911d-4a99-495d-bcf4-e6a3388bee56 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1ee7929b0aef45e18128b6aa2ea7db96'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:40:36.544138 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1195/2392] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:40:36 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 266 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 21:40:36.553309 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-f1aa911d-4a99-495d-bcf4-e6a3388bee56 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1ee7929b0aef45e18128b6aa2ea7db96'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:40:36.554148 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-f1aa911d-4a99-495d-bcf4-e6a3388bee56 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1ee7929b0aef45e18128b6aa2ea7db96'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:40:36.554766 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-f1aa911d-4a99-495d-bcf4-e6a3388bee56 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1ee7929b0aef45e18128b6aa2ea7db96'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:40:36.561984 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-f1aa911d-4a99-495d-bcf4-e6a3388bee56 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1ee7929b0aef45e18128b6aa2ea7db96'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:40:36.586010 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1198/2393] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:40:36 2022] DELETE /identity/v3/users/1ee7929b0aef45e18128b6aa2ea7db96 => generated 0 bytes in 207 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 21:40:36.797212 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e75a443f-a83f-45c3-9180-1d07ab868d8e service nova] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:40:36.799485 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-e75a443f-a83f-45c3-9180-1d07ab868d8e service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:40:36.819902 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e75a443f-a83f-45c3-9180-1d07ab868d8e service nova] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:40:36.930229 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-105f47bb-1f64-43eb-8999-23c54efc7bd6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:40:36.930994 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-105f47bb-1f64-43eb-8999-23c54efc7bd6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:40:36.931346 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-105f47bb-1f64-43eb-8999-23c54efc7bd6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:40:36.931766 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-105f47bb-1f64-43eb-8999-23c54efc7bd6 admin admin] PATH_INFO: `/v3/users/3be7a2a7ca204a5aa38907b9d08cdf7b` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:40:36.935119 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-105f47bb-1f64-43eb-8999-23c54efc7bd6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3be7a2a7ca204a5aa38907b9d08cdf7b)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:40:36.938178 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-105f47bb-1f64-43eb-8999-23c54efc7bd6 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:40:37.012775 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-105f47bb-1f64-43eb-8999-23c54efc7bd6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3be7a2a7ca204a5aa38907b9d08cdf7b'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:40:37.018080 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1aa911d-4a99-495d-bcf4-e6a3388bee56 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:40:37.019865 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-f1aa911d-4a99-495d-bcf4-e6a3388bee56 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:40:37.022073 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=93,keystone:DELETE=7 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:40:37.025656 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-105f47bb-1f64-43eb-8999-23c54efc7bd6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3be7a2a7ca204a5aa38907b9d08cdf7b'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:40:37.042483 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-105f47bb-1f64-43eb-8999-23c54efc7bd6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3be7a2a7ca204a5aa38907b9d08cdf7b'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:40:37.043743 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-105f47bb-1f64-43eb-8999-23c54efc7bd6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3be7a2a7ca204a5aa38907b9d08cdf7b'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:40:37.044621 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-105f47bb-1f64-43eb-8999-23c54efc7bd6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3be7a2a7ca204a5aa38907b9d08cdf7b'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:40:37.045250 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1aa911d-4a99-495d-bcf4-e6a3388bee56 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:40:37.053821 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-def112f9-0f2b-4536-814f-0ecf24c99ae4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:40:37.055198 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-def112f9-0f2b-4536-814f-0ecf24c99ae4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:40:37.055691 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-def112f9-0f2b-4536-814f-0ecf24c99ae4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:40:37.056172 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-def112f9-0f2b-4536-814f-0ecf24c99ae4 admin admin] PATH_INFO: `/v3/projects/c6b34827f8484817b85e0f94880e5e2e` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:40:37.058152 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-def112f9-0f2b-4536-814f-0ecf24c99ae4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c6b34827f8484817b85e0f94880e5e2e)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:40:37.060852 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-def112f9-0f2b-4536-814f-0ecf24c99ae4 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:40:37.074356 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1196/2394] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:40:36 2022] DELETE /identity/v3/users/3be7a2a7ca204a5aa38907b9d08cdf7b => generated 0 bytes in 275 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:40:37.075719 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b40e959-8ba2-47c3-af21-c4eb9d6d56cd None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:40:37.076271 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b40e959-8ba2-47c3-af21-c4eb9d6d56cd None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:40:37.076807 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b40e959-8ba2-47c3-af21-c4eb9d6d56cd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:40:37.104207 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-7b40e959-8ba2-47c3-af21-c4eb9d6d56cd None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:40:37.104841 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-7b40e959-8ba2-47c3-af21-c4eb9d6d56cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:40:37.120816 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:40:37.128899 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-def112f9-0f2b-4536-814f-0ecf24c99ae4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c6b34827f8484817b85e0f94880e5e2e', 'request_id': 'req-def112f9-0f2b-4536-814f-0ecf24c99ae4'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:40:37.140596 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-def112f9-0f2b-4536-814f-0ecf24c99ae4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c6b34827f8484817b85e0f94880e5e2e', 'request_id': 'req-def112f9-0f2b-4536-814f-0ecf24c99ae4'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:40:37.148231 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-def112f9-0f2b-4536-814f-0ecf24c99ae4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c6b34827f8484817b85e0f94880e5e2e', 'request_id': 'req-def112f9-0f2b-4536-814f-0ecf24c99ae4'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:40:37.167453 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-def112f9-0f2b-4536-814f-0ecf24c99ae4 admin admin] The token cache is being invalidate because project c6b34827f8484817b85e0f94880e5e2e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:40:37.167873 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-def112f9-0f2b-4536-814f-0ecf24c99ae4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:40:37.168619 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-def112f9-0f2b-4536-814f-0ecf24c99ae4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:40:37.170529 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1199/2395] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:40:37 2022] DELETE /identity/v3/projects/c6b34827f8484817b85e0f94880e5e2e => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 21:40:37.178164 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3da0b270-6761-4285-927a-9ca568ce5ca3 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:40:37.178474 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3da0b270-6761-4285-927a-9ca568ce5ca3 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:40:37.178758 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3da0b270-6761-4285-927a-9ca568ce5ca3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:40:37.199517 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1197/2396] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:40:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 21:40:37.204551 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-3da0b270-6761-4285-927a-9ca568ce5ca3 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:40:37.205942 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-3da0b270-6761-4285-927a-9ca568ce5ca3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:40:37.211076 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b40e959-8ba2-47c3-af21-c4eb9d6d56cd None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:40:37.212799 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-7b40e959-8ba2-47c3-af21-c4eb9d6d56cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:40:37.227064 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1200/2397] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:40:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 21:40:37.232231 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eea2a44-36ea-419d-8d4e-d94ca965e93c None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:40:37.232603 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eea2a44-36ea-419d-8d4e-d94ca965e93c None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:40:37.232786 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eea2a44-36ea-419d-8d4e-d94ca965e93c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:40:37.285439 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-3eea2a44-36ea-419d-8d4e-d94ca965e93c None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:40:37.287819 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-3eea2a44-36ea-419d-8d4e-d94ca965e93c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:40:37.310630 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1201/2398] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:40:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 77 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:40:37.312716 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eea2a44-36ea-419d-8d4e-d94ca965e93c None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:40:37.313318 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b40e959-8ba2-47c3-af21-c4eb9d6d56cd None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:40:37.326164 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eea2a44-36ea-419d-8d4e-d94ca965e93c None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:40:37.330769 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5da63427-a696-40fa-9296-10e2f9446394 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:40:37.332582 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da63427-a696-40fa-9296-10e2f9446394 service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:40:37.332782 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-704fc25a-46f2-4078-82b1-9d0907c6cddd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:40:37.332918 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da63427-a696-40fa-9296-10e2f9446394 service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:40:37.333125 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da63427-a696-40fa-9296-10e2f9446394 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:40:37.334046 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-704fc25a-46f2-4078-82b1-9d0907c6cddd admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:40:37.334428 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-704fc25a-46f2-4078-82b1-9d0907c6cddd admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:40:37.334961 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-704fc25a-46f2-4078-82b1-9d0907c6cddd admin admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:40:37.336302 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-704fc25a-46f2-4078-82b1-9d0907c6cddd admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:40:37.336820 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-704fc25a-46f2-4078-82b1-9d0907c6cddd admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:40:37.339420 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-704fc25a-46f2-4078-82b1-9d0907c6cddd admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:40:37.351439 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5da63427-a696-40fa-9296-10e2f9446394 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:40:37.360212 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5da63427-a696-40fa-9296-10e2f9446394 service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:40:37.366377 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1202/2399] 10.222.0.90 () {58 vars in 1153 bytes} [Mon Aug 29 21:40:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:40:37.371790 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-704fc25a-46f2-4078-82b1-9d0907c6cddd admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:40:37.374131 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1198/2400] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:40:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:40:37.383491 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-704fc25a-46f2-4078-82b1-9d0907c6cddd admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:40:37.392928 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02ebc974-962b-453f-bd7d-e64170a3067d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:40:37.393570 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02ebc974-962b-453f-bd7d-e64170a3067d admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:40:37.393894 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02ebc974-962b-453f-bd7d-e64170a3067d admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:40:37.394932 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02ebc974-962b-453f-bd7d-e64170a3067d admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:40:37.395795 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02ebc974-962b-453f-bd7d-e64170a3067d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:40:37.398607 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02ebc974-962b-453f-bd7d-e64170a3067d admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:40:37.440467 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1203/2401] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:40:37 2022] POST /identity/v3/projects => generated 377 bytes in 69 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 21:40:37.445385 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5da63427-a696-40fa-9296-10e2f9446394 service neutron] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:40:37.458049 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5da63427-a696-40fa-9296-10e2f9446394 service neutron] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:40:37.579978 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0a98d74-ddae-4758-93ce-3fdb1a7e1a27 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:40:37.580728 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0a98d74-ddae-4758-93ce-3fdb1a7e1a27 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:40:37.581010 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0a98d74-ddae-4758-93ce-3fdb1a7e1a27 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:40:37.581353 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0a98d74-ddae-4758-93ce-3fdb1a7e1a27 admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:40:37.582298 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0a98d74-ddae-4758-93ce-3fdb1a7e1a27 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:40:37.585161 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0a98d74-ddae-4758-93ce-3fdb1a7e1a27 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:40:37.621888 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1199/2402] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:40:37 2022] POST /identity/v3/users => generated 289 bytes in 177 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:40:37.625799 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02ebc974-962b-453f-bd7d-e64170a3067d admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:40:37.638038 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02ebc974-962b-453f-bd7d-e64170a3067d admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:40:37.646378 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2275a250-f65d-454d-9413-0a5ccdf7008d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:40:37.647123 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2275a250-f65d-454d-9413-0a5ccdf7008d admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:40:37.647313 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2275a250-f65d-454d-9413-0a5ccdf7008d admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:40:37.647600 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2275a250-f65d-454d-9413-0a5ccdf7008d admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:40:37.648270 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2275a250-f65d-454d-9413-0a5ccdf7008d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:40:37.650020 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2275a250-f65d-454d-9413-0a5ccdf7008d admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:40:37.672067 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1204/2403] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:40:37 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:40:37.673421 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0a98d74-ddae-4758-93ce-3fdb1a7e1a27 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:40:37.679389 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=93,keystone:INSERT=4,keystone:DELETE=3 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:40:37.687526 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0a98d74-ddae-4758-93ce-3fdb1a7e1a27 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:40:37.701984 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c79429a-8f1e-4cdb-a571-8154538f46cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:40:37.702712 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c79429a-8f1e-4cdb-a571-8154538f46cb admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:40:37.703067 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c79429a-8f1e-4cdb-a571-8154538f46cb admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:40:37.703351 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c79429a-8f1e-4cdb-a571-8154538f46cb admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:40:37.704334 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c79429a-8f1e-4cdb-a571-8154538f46cb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:40:37.706128 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c79429a-8f1e-4cdb-a571-8154538f46cb admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:40:37.728193 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2275a250-f65d-454d-9413-0a5ccdf7008d admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:40:37.730204 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-2275a250-f65d-454d-9413-0a5ccdf7008d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:40:37.730516 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1200/2404] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:40:37 2022] GET /identity/v3/roles => generated 1420 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:40:37.731920 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c79429a-8f1e-4cdb-a571-8154538f46cb admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:40:37.745042 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c79429a-8f1e-4cdb-a571-8154538f46cb admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:40:37.746254 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2275a250-f65d-454d-9413-0a5ccdf7008d admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:40:37.752009 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-630ac6c2-713b-40ba-95e9-72ee58d9cc9e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:40:37.753032 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-630ac6c2-713b-40ba-95e9-72ee58d9cc9e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:40:37.754180 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-630ac6c2-713b-40ba-95e9-72ee58d9cc9e admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:40:37.754766 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-630ac6c2-713b-40ba-95e9-72ee58d9cc9e admin admin] PATH_INFO: `/v3/projects/1c8fe22edbdd4ab5b4006dc5c526ec5f` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:40:37.756546 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-630ac6c2-713b-40ba-95e9-72ee58d9cc9e admin admin] RBAC: Authorizing `identity:delete_project(project_id=1c8fe22edbdd4ab5b4006dc5c526ec5f)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:40:37.758681 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-630ac6c2-713b-40ba-95e9-72ee58d9cc9e admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:40:37.760902 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f8016f1-e76e-43af-b0e1-390a8085a980 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:40:37.760902 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f8016f1-e76e-43af-b0e1-390a8085a980 admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:40:37.760902 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f8016f1-e76e-43af-b0e1-390a8085a980 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:40:37.760902 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f8016f1-e76e-43af-b0e1-390a8085a980 admin admin] PATH_INFO: `/v3/projects/0271b2ec720a402da3fcaad65ce4f7fd/users/2cc77a1abc0d45a38ef0bdc358e0e9ce/roles/02e242d5520740d4b77a5f4008c7eb57` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:40:37.785445 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f8016f1-e76e-43af-b0e1-390a8085a980 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0271b2ec720a402da3fcaad65ce4f7fd, user_id=2cc77a1abc0d45a38ef0bdc358e0e9ce, role_id=02e242d5520740d4b77a5f4008c7eb57)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:40:37.788931 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f8016f1-e76e-43af-b0e1-390a8085a980 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:40:37.807157 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1201/2405] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:40:37 2022] PUT /identity/v3/projects/0271b2ec720a402da3fcaad65ce4f7fd/users/2cc77a1abc0d45a38ef0bdc358e0e9ce/roles/02e242d5520740d4b77a5f4008c7eb57 => generated 0 bytes in 76 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:40:37.821148 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdbe0c7a-9c74-42be-bd5b-f5a4ae648b9a None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:40:37.821716 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdbe0c7a-9c74-42be-bd5b-f5a4ae648b9a None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:40:37.822033 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdbe0c7a-9c74-42be-bd5b-f5a4ae648b9a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:40:37.832766 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-630ac6c2-713b-40ba-95e9-72ee58d9cc9e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1c8fe22edbdd4ab5b4006dc5c526ec5f', 'request_id': 'req-630ac6c2-713b-40ba-95e9-72ee58d9cc9e'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:40:37.841830 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-630ac6c2-713b-40ba-95e9-72ee58d9cc9e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1c8fe22edbdd4ab5b4006dc5c526ec5f', 'request_id': 'req-630ac6c2-713b-40ba-95e9-72ee58d9cc9e'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:40:37.849697 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-630ac6c2-713b-40ba-95e9-72ee58d9cc9e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1c8fe22edbdd4ab5b4006dc5c526ec5f', 'request_id': 'req-630ac6c2-713b-40ba-95e9-72ee58d9cc9e'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:40:37.863955 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-fdbe0c7a-9c74-42be-bd5b-f5a4ae648b9a None None] MFA Rules not processed for user `2cc77a1abc0d45a38ef0bdc358e0e9ce`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:40:37.866490 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-fdbe0c7a-9c74-42be-bd5b-f5a4ae648b9a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:40:37.868518 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-630ac6c2-713b-40ba-95e9-72ee58d9cc9e admin admin] The token cache is being invalidate because project 1c8fe22edbdd4ab5b4006dc5c526ec5f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:40:37.868816 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-630ac6c2-713b-40ba-95e9-72ee58d9cc9e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:40:37.869478 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-630ac6c2-713b-40ba-95e9-72ee58d9cc9e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:40:37.871070 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1205/2406] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:40:37 2022] DELETE /identity/v3/projects/1c8fe22edbdd4ab5b4006dc5c526ec5f => generated 0 bytes in 143 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 21:40:37.875889 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a79614-71c6-4cb2-a26b-8951cc1c8f0e None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:40:37.876301 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a79614-71c6-4cb2-a26b-8951cc1c8f0e None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:40:37.876583 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a79614-71c6-4cb2-a26b-8951cc1c8f0e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:40:37.902853 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-85a79614-71c6-4cb2-a26b-8951cc1c8f0e None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:40:37.904329 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-85a79614-71c6-4cb2-a26b-8951cc1c8f0e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:40:37.963996 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1202/2407] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:40:37 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 21:40:37.966286 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdbe0c7a-9c74-42be-bd5b-f5a4ae648b9a None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:40:37.967923 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-fdbe0c7a-9c74-42be-bd5b-f5a4ae648b9a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:40:37.969650 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=86,keystone:DELETE=11,keystone:INSERT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:40:37.993209 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1206/2408] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:40:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 21:40:38.002691 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b210e9c7-78a8-45f6-9b83-2abb068d3bfd None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:40:38.003313 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b210e9c7-78a8-45f6-9b83-2abb068d3bfd None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:40:38.004405 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b210e9c7-78a8-45f6-9b83-2abb068d3bfd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:40:38.044072 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-b210e9c7-78a8-45f6-9b83-2abb068d3bfd None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:40:38.045405 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-b210e9c7-78a8-45f6-9b83-2abb068d3bfd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:40:38.055054 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1207/2409] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:40:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 54 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:40:38.059199 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b210e9c7-78a8-45f6-9b83-2abb068d3bfd None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:40:38.069895 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b210e9c7-78a8-45f6-9b83-2abb068d3bfd None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:40:38.075183 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-548dfe3e-2d3c-4673-83b2-5d397ca360ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:40:38.075815 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-548dfe3e-2d3c-4673-83b2-5d397ca360ac admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:40:38.076097 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-548dfe3e-2d3c-4673-83b2-5d397ca360ac admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:40:38.076366 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-548dfe3e-2d3c-4673-83b2-5d397ca360ac admin admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:40:38.077055 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-548dfe3e-2d3c-4673-83b2-5d397ca360ac admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:40:38.077327 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-548dfe3e-2d3c-4673-83b2-5d397ca360ac admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:40:38.079306 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-548dfe3e-2d3c-4673-83b2-5d397ca360ac admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:40:38.085272 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdbe0c7a-9c74-42be-bd5b-f5a4ae648b9a None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:40:38.093366 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80bb04f0-8d56-4bbe-a8bb-9eb99a13889c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:40:38.094031 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80bb04f0-8d56-4bbe-a8bb-9eb99a13889c admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:40:38.094309 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80bb04f0-8d56-4bbe-a8bb-9eb99a13889c admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:40:38.094579 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80bb04f0-8d56-4bbe-a8bb-9eb99a13889c admin admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:40:38.095226 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80bb04f0-8d56-4bbe-a8bb-9eb99a13889c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:40:38.097985 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80bb04f0-8d56-4bbe-a8bb-9eb99a13889c admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:40:38.101106 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1208/2410] 10.222.0.90 () {58 vars in 1153 bytes} [Mon Aug 29 21:40:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:40:38.105535 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-548dfe3e-2d3c-4673-83b2-5d397ca360ac admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:40:38.120550 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-548dfe3e-2d3c-4673-83b2-5d397ca360ac admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:40:38.126709 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-738b9e9d-25d2-4bd2-80b1-38d64044f4be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:40:38.127425 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738b9e9d-25d2-4bd2-80b1-38d64044f4be admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:40:38.127708 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738b9e9d-25d2-4bd2-80b1-38d64044f4be admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:40:38.128041 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738b9e9d-25d2-4bd2-80b1-38d64044f4be admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:40:38.128869 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-738b9e9d-25d2-4bd2-80b1-38d64044f4be admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:40:38.131174 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-738b9e9d-25d2-4bd2-80b1-38d64044f4be admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:40:38.147462 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1203/2411] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:40:37 2022] POST /identity/v3/projects => generated 377 bytes in 181 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 21:40:38.153609 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80bb04f0-8d56-4bbe-a8bb-9eb99a13889c admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:40:38.181828 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1209/2412] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:40:38 2022] POST /identity/v3/projects => generated 365 bytes in 75 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 21:40:38.184907 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-738b9e9d-25d2-4bd2-80b1-38d64044f4be admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:40:38.215804 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:40:38.239648 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80bb04f0-8d56-4bbe-a8bb-9eb99a13889c admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:40:38.271028 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-738b9e9d-25d2-4bd2-80b1-38d64044f4be admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:40:38.276253 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1f457b9-0f1f-46c8-8dde-36403b00491f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:40:38.276875 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1f457b9-0f1f-46c8-8dde-36403b00491f admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:40:38.277151 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1f457b9-0f1f-46c8-8dde-36403b00491f admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:40:38.277421 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1f457b9-0f1f-46c8-8dde-36403b00491f admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:40:38.278180 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1f457b9-0f1f-46c8-8dde-36403b00491f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:40:38.280116 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1f457b9-0f1f-46c8-8dde-36403b00491f admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:40:38.307751 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1210/2413] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:40:38 2022] POST /identity/v3/users => generated 283 bytes in 120 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:40:38.309891 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1f457b9-0f1f-46c8-8dde-36403b00491f admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:40:38.311488 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e55da5d9-b143-44a4-a4fe-4c2b55128cbb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:40:38.312115 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e55da5d9-b143-44a4-a4fe-4c2b55128cbb admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:40:38.312399 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e55da5d9-b143-44a4-a4fe-4c2b55128cbb admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:40:38.312661 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e55da5d9-b143-44a4-a4fe-4c2b55128cbb admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:40:38.313307 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e55da5d9-b143-44a4-a4fe-4c2b55128cbb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:40:38.315290 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e55da5d9-b143-44a4-a4fe-4c2b55128cbb admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:40:38.321777 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1f457b9-0f1f-46c8-8dde-36403b00491f admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:40:38.326990 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b82011d-3060-4648-80fb-0d584255bf6b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:40:38.327658 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b82011d-3060-4648-80fb-0d584255bf6b admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:40:38.327919 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b82011d-3060-4648-80fb-0d584255bf6b admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:40:38.328217 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b82011d-3060-4648-80fb-0d584255bf6b admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:40:38.329171 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b82011d-3060-4648-80fb-0d584255bf6b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:40:38.330935 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b82011d-3060-4648-80fb-0d584255bf6b admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:40:38.338195 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1204/2414] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:40:38 2022] POST /identity/v3/users => generated 295 bytes in 186 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 21:40:38.342295 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1211/2415] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:40:38 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:40:38.345231 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e55da5d9-b143-44a4-a4fe-4c2b55128cbb admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:40:38.346127 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b82011d-3060-4648-80fb-0d584255bf6b admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:40:38.353345 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e55da5d9-b143-44a4-a4fe-4c2b55128cbb admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:40:38.356104 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b82011d-3060-4648-80fb-0d584255bf6b admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:40:38.358458 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29e6c97b-d48a-4cd5-aaf4-edddce60c8d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:40:38.359087 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29e6c97b-d48a-4cd5-aaf4-edddce60c8d6 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:40:38.359370 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29e6c97b-d48a-4cd5-aaf4-edddce60c8d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:40:38.359645 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29e6c97b-d48a-4cd5-aaf4-edddce60c8d6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:40:38.360299 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29e6c97b-d48a-4cd5-aaf4-edddce60c8d6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:40:38.360854 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fe648f7-f5c9-40b7-9411-abc650fcf754 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:40:38.361473 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fe648f7-f5c9-40b7-9411-abc650fcf754 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:40:38.361759 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fe648f7-f5c9-40b7-9411-abc650fcf754 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:40:38.361982 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29e6c97b-d48a-4cd5-aaf4-edddce60c8d6 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:40:38.362154 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fe648f7-f5c9-40b7-9411-abc650fcf754 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:40:38.363513 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fe648f7-f5c9-40b7-9411-abc650fcf754 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:40:38.365147 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fe648f7-f5c9-40b7-9411-abc650fcf754 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:40:38.375051 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1205/2416] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:40:38 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:40:38.376379 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1212/2417] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:40:38 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:40:38.380402 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29e6c97b-d48a-4cd5-aaf4-edddce60c8d6 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:40:38.382792 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fe648f7-f5c9-40b7-9411-abc650fcf754 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:40:38.396128 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29e6c97b-d48a-4cd5-aaf4-edddce60c8d6 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:40:38.402207 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fe648f7-f5c9-40b7-9411-abc650fcf754 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:40:38.403499 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5c7ffeb-1a55-463a-81c1-03ed086f4598 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:40:38.404192 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5c7ffeb-1a55-463a-81c1-03ed086f4598 admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:40:38.404483 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5c7ffeb-1a55-463a-81c1-03ed086f4598 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:40:38.404836 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5c7ffeb-1a55-463a-81c1-03ed086f4598 admin admin] PATH_INFO: `/v3/projects/16dc7391579b41edaf0988b56b6b53f8/users/a82964d4cd6e4872b99b5aac12b77ca1/roles/a6e562cdc96c4be691d638990f8403fd` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:40:38.407776 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17294f92-2ae1-418d-936d-7f12bd44df4a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:40:38.408861 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17294f92-2ae1-418d-936d-7f12bd44df4a admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:40:38.409128 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17294f92-2ae1-418d-936d-7f12bd44df4a admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:40:38.409404 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17294f92-2ae1-418d-936d-7f12bd44df4a admin admin] PATH_INFO: `/v3/projects/32033beac47147c1ac79bfbe2f3192dd/users/000a997e0784440abf2040aa6f40a620/roles/02e242d5520740d4b77a5f4008c7eb57` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:40:38.422866 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5c7ffeb-1a55-463a-81c1-03ed086f4598 admin admin] RBAC: Authorizing `identity:create_grant(project_id=16dc7391579b41edaf0988b56b6b53f8, user_id=a82964d4cd6e4872b99b5aac12b77ca1, role_id=a6e562cdc96c4be691d638990f8403fd)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:40:38.426264 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5c7ffeb-1a55-463a-81c1-03ed086f4598 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:40:38.429263 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17294f92-2ae1-418d-936d-7f12bd44df4a admin admin] RBAC: Authorizing `identity:create_grant(project_id=32033beac47147c1ac79bfbe2f3192dd, user_id=000a997e0784440abf2040aa6f40a620, role_id=02e242d5520740d4b77a5f4008c7eb57)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:40:38.432237 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17294f92-2ae1-418d-936d-7f12bd44df4a admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:40:38.446110 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1206/2418] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:40:38 2022] PUT /identity/v3/projects/16dc7391579b41edaf0988b56b6b53f8/users/a82964d4cd6e4872b99b5aac12b77ca1/roles/a6e562cdc96c4be691d638990f8403fd => generated 0 bytes in 67 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:40:38.449834 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5c7ffeb-1a55-463a-81c1-03ed086f4598 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:40:38.452321 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1213/2419] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:40:38 2022] PUT /identity/v3/projects/32033beac47147c1ac79bfbe2f3192dd/users/000a997e0784440abf2040aa6f40a620/roles/02e242d5520740d4b77a5f4008c7eb57 => generated 0 bytes in 70 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:40:38.463167 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f57dba54-1962-455a-9d59-24c85f397077 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:40:38.463455 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f57dba54-1962-455a-9d59-24c85f397077 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:40:38.464217 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f57dba54-1962-455a-9d59-24c85f397077 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:40:38.504229 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-f57dba54-1962-455a-9d59-24c85f397077 None None] MFA Rules not processed for user `000a997e0784440abf2040aa6f40a620`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:40:38.506600 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-f57dba54-1962-455a-9d59-24c85f397077 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:40:38.538154 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5c7ffeb-1a55-463a-81c1-03ed086f4598 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:40:38.576558 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:40:38.618169 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1214/2420] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:40:38 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 152 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:40:38.618380 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:40:38.619595 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f57dba54-1962-455a-9d59-24c85f397077 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:40:38.640451 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17d700fb-a86a-46b4-b4e0-1f06628ff3c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:40:38.642472 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17d700fb-a86a-46b4-b4e0-1f06628ff3c0 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:40:38.643334 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17d700fb-a86a-46b4-b4e0-1f06628ff3c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:40:38.643626 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17d700fb-a86a-46b4-b4e0-1f06628ff3c0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:40:38.644345 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17d700fb-a86a-46b4-b4e0-1f06628ff3c0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:40:38.647855 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17d700fb-a86a-46b4-b4e0-1f06628ff3c0 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:40:38.667375 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1207/2421] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:40:38 2022] GET /identity/v3/roles => generated 1420 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:40:38.667375 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17d700fb-a86a-46b4-b4e0-1f06628ff3c0 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:40:38.676543 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17d700fb-a86a-46b4-b4e0-1f06628ff3c0 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:40:38.682576 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-993754fc-9b69-4637-b057-05744ca20fc3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:40:38.683323 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-993754fc-9b69-4637-b057-05744ca20fc3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:40:38.683626 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-993754fc-9b69-4637-b057-05744ca20fc3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:40:38.683902 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-993754fc-9b69-4637-b057-05744ca20fc3 admin admin] PATH_INFO: `/v3/projects/16dc7391579b41edaf0988b56b6b53f8/users/a82964d4cd6e4872b99b5aac12b77ca1/roles/a6e562cdc96c4be691d638990f8403fd` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:40:38.685913 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-993754fc-9b69-4637-b057-05744ca20fc3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=16dc7391579b41edaf0988b56b6b53f8, user_id=a82964d4cd6e4872b99b5aac12b77ca1, role_id=a6e562cdc96c4be691d638990f8403fd)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:40:38.688676 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-993754fc-9b69-4637-b057-05744ca20fc3 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:40:38.706241 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1208/2422] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:40:38 2022] PUT /identity/v3/projects/16dc7391579b41edaf0988b56b6b53f8/users/a82964d4cd6e4872b99b5aac12b77ca1/roles/a6e562cdc96c4be691d638990f8403fd => generated 0 bytes in 41 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:40:38.714188 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f57dba54-1962-455a-9d59-24c85f397077 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:40:38.716398 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be592f21-6630-4c37-8d4a-22645336abbb None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:40:38.716691 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be592f21-6630-4c37-8d4a-22645336abbb None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:40:38.716985 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be592f21-6630-4c37-8d4a-22645336abbb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:40:38.768714 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-be592f21-6630-4c37-8d4a-22645336abbb None None] MFA Rules not processed for user `a82964d4cd6e4872b99b5aac12b77ca1`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:40:38.769617 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-be592f21-6630-4c37-8d4a-22645336abbb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:40:38.814544 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5510d874-9f91-436f-8a89-25b2e4196885 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:40:38.815197 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5510d874-9f91-436f-8a89-25b2e4196885 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:40:38.815481 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5510d874-9f91-436f-8a89-25b2e4196885 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:40:38.815754 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5510d874-9f91-436f-8a89-25b2e4196885 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:40:38.816655 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5510d874-9f91-436f-8a89-25b2e4196885 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:40:38.818739 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5510d874-9f91-436f-8a89-25b2e4196885 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:40:38.864909 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1215/2423] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:40:38 2022] POST /identity/v3/projects => generated 363 bytes in 243 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:40:38.866522 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5510d874-9f91-436f-8a89-25b2e4196885 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:40:38.869630 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1209/2424] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:40:38 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:40:38.884653 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be592f21-6630-4c37-8d4a-22645336abbb None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:40:38.886113 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-be592f21-6630-4c37-8d4a-22645336abbb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:40:38.938977 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5510d874-9f91-436f-8a89-25b2e4196885 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:40:38.944447 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a410391-3d31-4751-ad1c-3cc13416c776 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:40:38.944813 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a410391-3d31-4751-ad1c-3cc13416c776 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:40:38.945091 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a410391-3d31-4751-ad1c-3cc13416c776 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:40:38.945368 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a410391-3d31-4751-ad1c-3cc13416c776 admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:40:38.946025 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a410391-3d31-4751-ad1c-3cc13416c776 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:40:38.947839 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be592f21-6630-4c37-8d4a-22645336abbb None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:40:38.948072 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a410391-3d31-4751-ad1c-3cc13416c776 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:40:38.972836 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1216/2425] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:40:38 2022] POST /identity/v3/users => generated 288 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 21:40:38.976940 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a410391-3d31-4751-ad1c-3cc13416c776 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:40:38.987730 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a410391-3d31-4751-ad1c-3cc13416c776 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:40:38.992793 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-396f78bb-76ea-45c2-a01f-17efc4a494f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:40:38.993405 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-396f78bb-76ea-45c2-a01f-17efc4a494f3 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:40:38.993685 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-396f78bb-76ea-45c2-a01f-17efc4a494f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:40:38.993998 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-396f78bb-76ea-45c2-a01f-17efc4a494f3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:40:38.994642 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-396f78bb-76ea-45c2-a01f-17efc4a494f3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:40:38.996237 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-396f78bb-76ea-45c2-a01f-17efc4a494f3 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:40:38.998932 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:40:39.003999 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-441a91eb-7105-4b0e-99c0-7320696f7546 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dc226f7ddcb49ecacb51ce57fdfc11b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:40:39.004629 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-441a91eb-7105-4b0e-99c0-7320696f7546 service cinder] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:40:39.004907 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-441a91eb-7105-4b0e-99c0-7320696f7546 service cinder] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:40:39.005181 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-441a91eb-7105-4b0e-99c0-7320696f7546 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:40:39.014228 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1217/2426] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:40:38 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 29 21:40:39.015441 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-396f78bb-76ea-45c2-a01f-17efc4a494f3 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:40:39.026702 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-396f78bb-76ea-45c2-a01f-17efc4a494f3 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:40:39.031888 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f08500b0-d3b8-485e-b88b-4e75bc2b46d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:40:39.032565 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f08500b0-d3b8-485e-b88b-4e75bc2b46d2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:40:39.032850 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f08500b0-d3b8-485e-b88b-4e75bc2b46d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:40:39.033118 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f08500b0-d3b8-485e-b88b-4e75bc2b46d2 admin admin] PATH_INFO: `/v3/projects/95ec32538a63421f958b81ef1b2c2f12/users/9bbc7af011e54e6e9c9b4d23aab90e3b/roles/a6e562cdc96c4be691d638990f8403fd` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:40:39.042962 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:40:39.049694 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f08500b0-d3b8-485e-b88b-4e75bc2b46d2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=95ec32538a63421f958b81ef1b2c2f12, user_id=9bbc7af011e54e6e9c9b4d23aab90e3b, role_id=a6e562cdc96c4be691d638990f8403fd)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:40:39.052668 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f08500b0-d3b8-485e-b88b-4e75bc2b46d2 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:40:39.066771 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1218/2427] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:40:39 2022] PUT /identity/v3/projects/95ec32538a63421f958b81ef1b2c2f12/users/9bbc7af011e54e6e9c9b4d23aab90e3b/roles/a6e562cdc96c4be691d638990f8403fd => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:40:39.071648 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f08500b0-d3b8-485e-b88b-4e75bc2b46d2 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:40:39.079665 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-441a91eb-7105-4b0e-99c0-7320696f7546 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:40:39.081115 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-441a91eb-7105-4b0e-99c0-7320696f7546 service cinder] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:40:39.092211 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1210/2428] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:40:38 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 208 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 21:40:39.140149 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f08500b0-d3b8-485e-b88b-4e75bc2b46d2 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:40:39.146071 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c763ba2-7a04-4b14-9033-17d165fb8d45 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:40:39.146686 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c763ba2-7a04-4b14-9033-17d165fb8d45 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:40:39.146979 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c763ba2-7a04-4b14-9033-17d165fb8d45 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:40:39.147240 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c763ba2-7a04-4b14-9033-17d165fb8d45 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:40:39.147885 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c763ba2-7a04-4b14-9033-17d165fb8d45 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:40:39.149484 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c763ba2-7a04-4b14-9033-17d165fb8d45 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:40:39.183444 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1219/2429] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:40:39 2022] GET /identity/v3/roles => generated 1420 bytes in 112 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 29 21:40:39.188523 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-441a91eb-7105-4b0e-99c0-7320696f7546 service cinder] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:40:39.207299 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-441a91eb-7105-4b0e-99c0-7320696f7546 service cinder] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:40:39.217573 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f5eff50-2d6c-42e5-80e5-ab53a6e6a8f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:40:39.218993 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f5eff50-2d6c-42e5-80e5-ab53a6e6a8f5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:40:39.219417 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f5eff50-2d6c-42e5-80e5-ab53a6e6a8f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:40:39.219897 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f5eff50-2d6c-42e5-80e5-ab53a6e6a8f5 admin admin] PATH_INFO: `/v3/projects/95ec32538a63421f958b81ef1b2c2f12/users/9bbc7af011e54e6e9c9b4d23aab90e3b/roles/a6e562cdc96c4be691d638990f8403fd` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:40:39.224451 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f5eff50-2d6c-42e5-80e5-ab53a6e6a8f5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=95ec32538a63421f958b81ef1b2c2f12, user_id=9bbc7af011e54e6e9c9b4d23aab90e3b, role_id=a6e562cdc96c4be691d638990f8403fd)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:40:39.229401 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f5eff50-2d6c-42e5-80e5-ab53a6e6a8f5 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:40:39.248174 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1211/2430] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:40:39 2022] PUT /identity/v3/projects/95ec32538a63421f958b81ef1b2c2f12/users/9bbc7af011e54e6e9c9b4d23aab90e3b/roles/a6e562cdc96c4be691d638990f8403fd => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 21:40:39.261260 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e64d361b-2472-4f82-825c-61fff9e164a9 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:40:39.261541 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e64d361b-2472-4f82-825c-61fff9e164a9 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:40:39.261777 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e64d361b-2472-4f82-825c-61fff9e164a9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:40:39.298679 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-e64d361b-2472-4f82-825c-61fff9e164a9 None None] MFA Rules not processed for user `9bbc7af011e54e6e9c9b4d23aab90e3b`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:40:39.300175 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-e64d361b-2472-4f82-825c-61fff9e164a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:40:39.405556 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1220/2431] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:40:39 2022] POST /identity/v3/auth/tokens => generated 3307 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:40:39.418333 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f5eff50-2d6c-42e5-80e5-ab53a6e6a8f5 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:40:39.521767 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f5eff50-2d6c-42e5-80e5-ab53a6e6a8f5 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:40:39.526664 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26c59c96-ee61-443f-b884-924ee0962a39 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dc226f7ddcb49ecacb51ce57fdfc11b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:40:39.527419 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26c59c96-ee61-443f-b884-924ee0962a39 service cinder] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:40:39.527792 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26c59c96-ee61-443f-b884-924ee0962a39 service cinder] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:40:39.528138 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26c59c96-ee61-443f-b884-924ee0962a39 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:40:39.539244 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26c59c96-ee61-443f-b884-924ee0962a39 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:40:39.540756 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26c59c96-ee61-443f-b884-924ee0962a39 service cinder] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:40:39.553706 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1212/2432] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:40:39 2022] GET /identity/v3/auth/tokens => generated 3307 bytes in 136 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:40:39.647223 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e64d361b-2472-4f82-825c-61fff9e164a9 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:40:39.659475 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e64d361b-2472-4f82-825c-61fff9e164a9 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:40:39.665729 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fc08565-147a-4ad7-af40-de673ab20537 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dc226f7ddcb49ecacb51ce57fdfc11b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:40:39.666390 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc08565-147a-4ad7-af40-de673ab20537 service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:40:39.666676 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc08565-147a-4ad7-af40-de673ab20537 service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:40:39.666959 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc08565-147a-4ad7-af40-de673ab20537 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:40:39.730822 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:40:39.736471 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fc08565-147a-4ad7-af40-de673ab20537 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:40:39.739019 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fc08565-147a-4ad7-af40-de673ab20537 service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:40:39.752363 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1221/2433] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:40:39 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:40:40.375595 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aa072f6-6923-47e3-bde0-33a111213338 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:40:40.376247 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aa072f6-6923-47e3-bde0-33a111213338 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:40:40.376247 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aa072f6-6923-47e3-bde0-33a111213338 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:40:40.399083 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-2aa072f6-6923-47e3-bde0-33a111213338 None None] MFA Rules not processed for user `0c0b0da87b124de1be00faec4bbdb404`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:40:40.400296 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-2aa072f6-6923-47e3-bde0-33a111213338 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:40:40.464598 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1213/2434] 10.222.0.90 () {58 vars in 1009 bytes} [Mon Aug 29 21:40:40 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:40:40.477550 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fc08565-147a-4ad7-af40-de673ab20537 service cinder] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:40:40.479340 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-5fc08565-147a-4ad7-af40-de673ab20537 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:40:40.563042 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fc08565-147a-4ad7-af40-de673ab20537 service cinder] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:40:40.570111 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36f45b5b-75b3-4995-9604-da806a43f221 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '139bbbde6d884b42b5440727926e2c58', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:40:40.570877 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36f45b5b-75b3-4995-9604-da806a43f221 service swift] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:40:40.571271 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36f45b5b-75b3-4995-9604-da806a43f221 service swift] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:40:40.571673 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36f45b5b-75b3-4995-9604-da806a43f221 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:40:40.582796 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36f45b5b-75b3-4995-9604-da806a43f221 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:40:40.584595 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36f45b5b-75b3-4995-9604-da806a43f221 service swift] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:40:40.594952 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1222/2435] 10.222.0.90 () {60 vars in 1384 bytes} [Mon Aug 29 21:40:40 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 118 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Aug 29 21:40:43.949089 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aa072f6-6923-47e3-bde0-33a111213338 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:40:43.953317 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-2aa072f6-6923-47e3-bde0-33a111213338 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:40:43.978387 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aa072f6-6923-47e3-bde0-33a111213338 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:40:43.986978 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70413852-75e1-48ea-8a25-cc6cbb08abe8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dc226f7ddcb49ecacb51ce57fdfc11b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:40:43.989933 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70413852-75e1-48ea-8a25-cc6cbb08abe8 service cinder] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:40:43.990586 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70413852-75e1-48ea-8a25-cc6cbb08abe8 service cinder] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:40:43.991179 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70413852-75e1-48ea-8a25-cc6cbb08abe8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:40:44.111982 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70413852-75e1-48ea-8a25-cc6cbb08abe8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:40:44.114945 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70413852-75e1-48ea-8a25-cc6cbb08abe8 service cinder] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:40:44.120233 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:40:44.134541 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1214/2436] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:40:43 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 186 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:40:50.591657 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=35 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:40:54.138293 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:40:57.470892 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36f45b5b-75b3-4995-9604-da806a43f221 service swift] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:40:57.547458 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36f45b5b-75b3-4995-9604-da806a43f221 service swift] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:40:57.552219 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-563c8938-034e-4642-94c3-10675ec311ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:40:57.553221 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-563c8938-034e-4642-94c3-10675ec311ff admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:40:57.553508 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-563c8938-034e-4642-94c3-10675ec311ff admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:40:57.553848 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-563c8938-034e-4642-94c3-10675ec311ff admin admin] PATH_INFO: `/v3/users/000a997e0784440abf2040aa6f40a620` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:40:57.554972 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-563c8938-034e-4642-94c3-10675ec311ff admin admin] RBAC: Authorizing `identity:delete_user(user_id=000a997e0784440abf2040aa6f40a620)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:40:57.556883 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-563c8938-034e-4642-94c3-10675ec311ff admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:40:57.609664 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-563c8938-034e-4642-94c3-10675ec311ff admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '000a997e0784440abf2040aa6f40a620'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:40:57.625497 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-563c8938-034e-4642-94c3-10675ec311ff admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '000a997e0784440abf2040aa6f40a620'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:40:57.626330 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-563c8938-034e-4642-94c3-10675ec311ff admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '000a997e0784440abf2040aa6f40a620'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:40:57.626951 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-563c8938-034e-4642-94c3-10675ec311ff admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '000a997e0784440abf2040aa6f40a620'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:40:57.639835 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-563c8938-034e-4642-94c3-10675ec311ff admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '000a997e0784440abf2040aa6f40a620'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:40:57.656931 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1223/2437] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:40:57 2022] DELETE /identity/v3/users/000a997e0784440abf2040aa6f40a620 => generated 0 bytes in 187 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 21:40:57.661538 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc7b9587-48c6-45d0-b63f-d3be1a56ed47 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:40:57.662202 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc7b9587-48c6-45d0-b63f-d3be1a56ed47 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:40:57.662202 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc7b9587-48c6-45d0-b63f-d3be1a56ed47 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:40:57.683450 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-bc7b9587-48c6-45d0-b63f-d3be1a56ed47 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:40:57.684757 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-bc7b9587-48c6-45d0-b63f-d3be1a56ed47 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:40:57.760042 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1215/2438] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:40:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:40:57.771612 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-563c8938-034e-4642-94c3-10675ec311ff admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:40:57.773236 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-563c8938-034e-4642-94c3-10675ec311ff admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:40:57.831463 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-563c8938-034e-4642-94c3-10675ec311ff admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:40:57.835152 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43197a39-98be-4036-9156-fe2b0d66daf2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:40:57.835764 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43197a39-98be-4036-9156-fe2b0d66daf2 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:40:57.836047 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43197a39-98be-4036-9156-fe2b0d66daf2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:40:57.836328 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43197a39-98be-4036-9156-fe2b0d66daf2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:40:57.845496 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43197a39-98be-4036-9156-fe2b0d66daf2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:40:57.846992 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43197a39-98be-4036-9156-fe2b0d66daf2 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:40:57.856840 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1224/2439] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:40:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:40:58.143815 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc7b9587-48c6-45d0-b63f-d3be1a56ed47 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:40:58.145566 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-bc7b9587-48c6-45d0-b63f-d3be1a56ed47 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:40:58.158140 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc7b9587-48c6-45d0-b63f-d3be1a56ed47 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:40:58.168177 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73469380-682b-413d-a4bf-8d30e79a0f9c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:40:58.168933 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73469380-682b-413d-a4bf-8d30e79a0f9c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:40:58.169321 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73469380-682b-413d-a4bf-8d30e79a0f9c admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:40:58.169660 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73469380-682b-413d-a4bf-8d30e79a0f9c admin admin] PATH_INFO: `/v3/projects/32033beac47147c1ac79bfbe2f3192dd` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:40:58.170930 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73469380-682b-413d-a4bf-8d30e79a0f9c admin admin] RBAC: Authorizing `identity:delete_project(project_id=32033beac47147c1ac79bfbe2f3192dd)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:40:58.173075 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73469380-682b-413d-a4bf-8d30e79a0f9c admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:40:58.225671 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-73469380-682b-413d-a4bf-8d30e79a0f9c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '32033beac47147c1ac79bfbe2f3192dd', 'request_id': 'req-73469380-682b-413d-a4bf-8d30e79a0f9c'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:40:58.231574 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-73469380-682b-413d-a4bf-8d30e79a0f9c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '32033beac47147c1ac79bfbe2f3192dd', 'request_id': 'req-73469380-682b-413d-a4bf-8d30e79a0f9c'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:40:58.243756 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-73469380-682b-413d-a4bf-8d30e79a0f9c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '32033beac47147c1ac79bfbe2f3192dd', 'request_id': 'req-73469380-682b-413d-a4bf-8d30e79a0f9c'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:40:58.259931 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-73469380-682b-413d-a4bf-8d30e79a0f9c admin admin] The token cache is being invalidate because project 32033beac47147c1ac79bfbe2f3192dd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81434) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:40:58.260231 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-73469380-682b-413d-a4bf-8d30e79a0f9c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:40:58.260958 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-73469380-682b-413d-a4bf-8d30e79a0f9c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:40:58.262531 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1216/2440] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:40:58 2022] DELETE /identity/v3/projects/32033beac47147c1ac79bfbe2f3192dd => generated 0 bytes in 120 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:40:58.266772 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43197a39-98be-4036-9156-fe2b0d66daf2 service neutron] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:40:58.278636 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43197a39-98be-4036-9156-fe2b0d66daf2 service neutron] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:40:58.298094 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:40:58.347532 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-832dfa55-3cd7-46e4-8624-c99c722501de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:40:58.348192 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-832dfa55-3cd7-46e4-8624-c99c722501de admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:40:58.348476 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-832dfa55-3cd7-46e4-8624-c99c722501de admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:40:58.348750 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-832dfa55-3cd7-46e4-8624-c99c722501de admin admin] PATH_INFO: `/v3/users/9bbc7af011e54e6e9c9b4d23aab90e3b` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:40:58.350244 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-832dfa55-3cd7-46e4-8624-c99c722501de admin admin] RBAC: Authorizing `identity:delete_user(user_id=9bbc7af011e54e6e9c9b4d23aab90e3b)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:40:58.352218 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-832dfa55-3cd7-46e4-8624-c99c722501de admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:40:58.409102 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-832dfa55-3cd7-46e4-8624-c99c722501de admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9bbc7af011e54e6e9c9b4d23aab90e3b'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:40:58.421192 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-832dfa55-3cd7-46e4-8624-c99c722501de admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9bbc7af011e54e6e9c9b4d23aab90e3b'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:40:58.421907 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-832dfa55-3cd7-46e4-8624-c99c722501de admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9bbc7af011e54e6e9c9b4d23aab90e3b'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:40:58.422531 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-832dfa55-3cd7-46e4-8624-c99c722501de admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9bbc7af011e54e6e9c9b4d23aab90e3b'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:40:58.429798 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-832dfa55-3cd7-46e4-8624-c99c722501de admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9bbc7af011e54e6e9c9b4d23aab90e3b'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:40:58.443434 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1225/2441] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:40:58 2022] DELETE /identity/v3/users/9bbc7af011e54e6e9c9b4d23aab90e3b => generated 0 bytes in 177 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:40:58.726848 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73469380-682b-413d-a4bf-8d30e79a0f9c admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:40:58.728379 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-73469380-682b-413d-a4bf-8d30e79a0f9c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:40:58.742924 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73469380-682b-413d-a4bf-8d30e79a0f9c admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:40:58.806855 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a8b13a6-0b95-48db-8285-af441f3e10f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:40:58.807493 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a8b13a6-0b95-48db-8285-af441f3e10f0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:40:58.807776 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a8b13a6-0b95-48db-8285-af441f3e10f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:40:58.808053 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a8b13a6-0b95-48db-8285-af441f3e10f0 admin admin] PATH_INFO: `/v3/projects/95ec32538a63421f958b81ef1b2c2f12` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:40:58.809183 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a8b13a6-0b95-48db-8285-af441f3e10f0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=95ec32538a63421f958b81ef1b2c2f12)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:40:58.811140 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a8b13a6-0b95-48db-8285-af441f3e10f0 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:40:58.846344 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:40:58.865403 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-0a8b13a6-0b95-48db-8285-af441f3e10f0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '95ec32538a63421f958b81ef1b2c2f12', 'request_id': 'req-0a8b13a6-0b95-48db-8285-af441f3e10f0'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:40:58.871332 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-0a8b13a6-0b95-48db-8285-af441f3e10f0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '95ec32538a63421f958b81ef1b2c2f12', 'request_id': 'req-0a8b13a6-0b95-48db-8285-af441f3e10f0'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:40:58.879703 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-0a8b13a6-0b95-48db-8285-af441f3e10f0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '95ec32538a63421f958b81ef1b2c2f12', 'request_id': 'req-0a8b13a6-0b95-48db-8285-af441f3e10f0'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:40:58.895852 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-0a8b13a6-0b95-48db-8285-af441f3e10f0 admin admin] The token cache is being invalidate because project 95ec32538a63421f958b81ef1b2c2f12 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81434) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:40:58.896348 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-0a8b13a6-0b95-48db-8285-af441f3e10f0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:40:58.897329 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-0a8b13a6-0b95-48db-8285-af441f3e10f0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:40:58.899527 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1217/2442] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:40:58 2022] DELETE /identity/v3/projects/95ec32538a63421f958b81ef1b2c2f12 => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:40:58.905757 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90447f41-c5f9-417f-aebb-5191de3b657d None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:40:58.906272 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90447f41-c5f9-417f-aebb-5191de3b657d None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:40:58.906384 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90447f41-c5f9-417f-aebb-5191de3b657d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:40:58.926580 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-90447f41-c5f9-417f-aebb-5191de3b657d None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:40:58.927779 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-90447f41-c5f9-417f-aebb-5191de3b657d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:40:58.988536 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1226/2443] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:40:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:40:58.996517 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01024885-5245-4472-8047-02b3df6a05f0 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:40:58.996733 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01024885-5245-4472-8047-02b3df6a05f0 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:40:58.997012 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01024885-5245-4472-8047-02b3df6a05f0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:40:59.025532 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-01024885-5245-4472-8047-02b3df6a05f0 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:40:59.026799 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-01024885-5245-4472-8047-02b3df6a05f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:40:59.034791 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1218/2444] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:40:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:40:59.039093 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90447f41-c5f9-417f-aebb-5191de3b657d None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:40:59.048641 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90447f41-c5f9-417f-aebb-5191de3b657d None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:40:59.053621 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47671e68-c1dc-4907-9be3-6e0b085caf0d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:40:59.054271 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47671e68-c1dc-4907-9be3-6e0b085caf0d admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:40:59.054556 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47671e68-c1dc-4907-9be3-6e0b085caf0d admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:40:59.054832 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47671e68-c1dc-4907-9be3-6e0b085caf0d admin admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:40:59.055507 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47671e68-c1dc-4907-9be3-6e0b085caf0d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:40:59.055815 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47671e68-c1dc-4907-9be3-6e0b085caf0d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:40:59.057495 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47671e68-c1dc-4907-9be3-6e0b085caf0d admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:40:59.069535 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1227/2445] 10.222.0.90 () {58 vars in 1153 bytes} [Mon Aug 29 21:40:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:40:59.074317 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01024885-5245-4472-8047-02b3df6a05f0 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:40:59.084647 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01024885-5245-4472-8047-02b3df6a05f0 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:40:59.089361 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c37bebf-7461-49c5-bb0c-ca00cf78c76a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:40:59.090002 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c37bebf-7461-49c5-bb0c-ca00cf78c76a admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:40:59.090287 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c37bebf-7461-49c5-bb0c-ca00cf78c76a admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:40:59.090560 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c37bebf-7461-49c5-bb0c-ca00cf78c76a admin admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:40:59.091233 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c37bebf-7461-49c5-bb0c-ca00cf78c76a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:40:59.093148 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c37bebf-7461-49c5-bb0c-ca00cf78c76a admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:40:59.128373 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1219/2446] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:40:59 2022] POST /identity/v3/projects => generated 383 bytes in 55 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:40:59.132771 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47671e68-c1dc-4907-9be3-6e0b085caf0d admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:40:59.142899 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47671e68-c1dc-4907-9be3-6e0b085caf0d admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:40:59.169939 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:40:59.203051 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7bf3a32-9f06-41fd-9f56-9eee9e70c651 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:40:59.203915 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7bf3a32-9f06-41fd-9f56-9eee9e70c651 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:40:59.204202 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7bf3a32-9f06-41fd-9f56-9eee9e70c651 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:40:59.204480 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7bf3a32-9f06-41fd-9f56-9eee9e70c651 admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:40:59.205128 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7bf3a32-9f06-41fd-9f56-9eee9e70c651 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:40:59.207380 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7bf3a32-9f06-41fd-9f56-9eee9e70c651 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:40:59.228820 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1228/2447] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:40:59 2022] POST /identity/v3/users => generated 292 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 21:40:59.233060 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c37bebf-7461-49c5-bb0c-ca00cf78c76a admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:40:59.244260 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c37bebf-7461-49c5-bb0c-ca00cf78c76a admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:40:59.249288 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9af0759d-b0e6-4653-a062-4162ec175511 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:40:59.249864 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9af0759d-b0e6-4653-a062-4162ec175511 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:40:59.250147 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9af0759d-b0e6-4653-a062-4162ec175511 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:40:59.250421 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9af0759d-b0e6-4653-a062-4162ec175511 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:40:59.251220 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9af0759d-b0e6-4653-a062-4162ec175511 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:40:59.253231 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9af0759d-b0e6-4653-a062-4162ec175511 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:40:59.266753 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1220/2448] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:40:59 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:40:59.268411 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7bf3a32-9f06-41fd-9f56-9eee9e70c651 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:40:59.281806 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7bf3a32-9f06-41fd-9f56-9eee9e70c651 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:40:59.287316 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ae4976c-cf0a-4a93-8f53-b7ec8d41cf52 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:40:59.287979 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ae4976c-cf0a-4a93-8f53-b7ec8d41cf52 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:40:59.288264 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ae4976c-cf0a-4a93-8f53-b7ec8d41cf52 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:40:59.288535 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ae4976c-cf0a-4a93-8f53-b7ec8d41cf52 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:40:59.289209 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ae4976c-cf0a-4a93-8f53-b7ec8d41cf52 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:40:59.290852 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ae4976c-cf0a-4a93-8f53-b7ec8d41cf52 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:40:59.303847 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1229/2449] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:40:59 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:40:59.308278 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9af0759d-b0e6-4653-a062-4162ec175511 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:40:59.320005 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9af0759d-b0e6-4653-a062-4162ec175511 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:40:59.325369 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbf5fe45-d2d0-4a79-a4f3-832fedf988f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:40:59.326243 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbf5fe45-d2d0-4a79-a4f3-832fedf988f7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:40:59.326562 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbf5fe45-d2d0-4a79-a4f3-832fedf988f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:40:59.326834 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbf5fe45-d2d0-4a79-a4f3-832fedf988f7 admin admin] PATH_INFO: `/v3/projects/0eb22bb4364e4999b0b0a399ff81c969/users/ea135e9ede274721b13dfa01f033aa7f/roles/02e242d5520740d4b77a5f4008c7eb57` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:40:59.346327 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbf5fe45-d2d0-4a79-a4f3-832fedf988f7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0eb22bb4364e4999b0b0a399ff81c969, user_id=ea135e9ede274721b13dfa01f033aa7f, role_id=02e242d5520740d4b77a5f4008c7eb57)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:40:59.348990 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbf5fe45-d2d0-4a79-a4f3-832fedf988f7 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:40:59.371918 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1221/2450] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:40:59 2022] PUT /identity/v3/projects/0eb22bb4364e4999b0b0a399ff81c969/users/ea135e9ede274721b13dfa01f033aa7f/roles/02e242d5520740d4b77a5f4008c7eb57 => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:40:59.373612 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eca4ae0-958e-4ae1-84b5-bc92551f55dd None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:40:59.373943 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eca4ae0-958e-4ae1-84b5-bc92551f55dd None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:40:59.374276 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eca4ae0-958e-4ae1-84b5-bc92551f55dd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:40:59.407836 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-4eca4ae0-958e-4ae1-84b5-bc92551f55dd None None] MFA Rules not processed for user `ea135e9ede274721b13dfa01f033aa7f`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:40:59.409404 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-4eca4ae0-958e-4ae1-84b5-bc92551f55dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:40:59.483060 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1230/2451] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:40:59 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:40:59.487674 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbf5fe45-d2d0-4a79-a4f3-832fedf988f7 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:40:59.561563 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbf5fe45-d2d0-4a79-a4f3-832fedf988f7 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:40:59.569742 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-197ff6a0-9be9-45c3-bbdc-609dbd46c2c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:40:59.570767 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-197ff6a0-9be9-45c3-bbdc-609dbd46c2c4 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:40:59.571102 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-197ff6a0-9be9-45c3-bbdc-609dbd46c2c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:40:59.571381 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-197ff6a0-9be9-45c3-bbdc-609dbd46c2c4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:40:59.572041 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-197ff6a0-9be9-45c3-bbdc-609dbd46c2c4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:40:59.573968 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-197ff6a0-9be9-45c3-bbdc-609dbd46c2c4 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:40:59.613841 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1222/2452] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:40:59 2022] POST /identity/v3/projects => generated 383 bytes in 123 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:40:59.615625 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eca4ae0-958e-4ae1-84b5-bc92551f55dd None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:40:59.631442 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eca4ae0-958e-4ae1-84b5-bc92551f55dd None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:40:59.701470 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e89ea0d3-2d04-44ea-a8cd-b0cbe76ee9a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:40:59.702227 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e89ea0d3-2d04-44ea-a8cd-b0cbe76ee9a1 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:40:59.702590 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e89ea0d3-2d04-44ea-a8cd-b0cbe76ee9a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:40:59.702943 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e89ea0d3-2d04-44ea-a8cd-b0cbe76ee9a1 admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:40:59.703986 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e89ea0d3-2d04-44ea-a8cd-b0cbe76ee9a1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:40:59.707242 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e89ea0d3-2d04-44ea-a8cd-b0cbe76ee9a1 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:40:59.733098 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1231/2453] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:40:59 2022] POST /identity/v3/users => generated 298 bytes in 118 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:40:59.737645 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-197ff6a0-9be9-45c3-bbdc-609dbd46c2c4 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:40:59.748298 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-197ff6a0-9be9-45c3-bbdc-609dbd46c2c4 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:40:59.754579 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-471b1f46-ab2d-40e9-9e51-f10f1aa7077c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:40:59.755203 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-471b1f46-ab2d-40e9-9e51-f10f1aa7077c admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:40:59.755487 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-471b1f46-ab2d-40e9-9e51-f10f1aa7077c admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:40:59.755759 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-471b1f46-ab2d-40e9-9e51-f10f1aa7077c admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:40:59.756408 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-471b1f46-ab2d-40e9-9e51-f10f1aa7077c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:40:59.758155 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-471b1f46-ab2d-40e9-9e51-f10f1aa7077c admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:40:59.770322 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1223/2454] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:40:59 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:40:59.774626 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e89ea0d3-2d04-44ea-a8cd-b0cbe76ee9a1 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:40:59.786081 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e89ea0d3-2d04-44ea-a8cd-b0cbe76ee9a1 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:40:59.790984 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d7b47cc-4019-4891-9ac4-432a402009e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:40:59.791755 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d7b47cc-4019-4891-9ac4-432a402009e6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:40:59.792145 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d7b47cc-4019-4891-9ac4-432a402009e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:40:59.792423 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d7b47cc-4019-4891-9ac4-432a402009e6 admin admin] PATH_INFO: `/v3/projects/355fe3e8b7754671aa90d66f1783ec01/users/66e0c0ed344440d1b93c961976feac67/roles/a6e562cdc96c4be691d638990f8403fd` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:40:59.807809 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d7b47cc-4019-4891-9ac4-432a402009e6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=355fe3e8b7754671aa90d66f1783ec01, user_id=66e0c0ed344440d1b93c961976feac67, role_id=a6e562cdc96c4be691d638990f8403fd)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:40:59.810652 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d7b47cc-4019-4891-9ac4-432a402009e6 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:40:59.825966 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1232/2455] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:40:59 2022] PUT /identity/v3/projects/355fe3e8b7754671aa90d66f1783ec01/users/66e0c0ed344440d1b93c961976feac67/roles/a6e562cdc96c4be691d638990f8403fd => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:40:59.831051 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-471b1f46-ab2d-40e9-9e51-f10f1aa7077c admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:40:59.841507 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-471b1f46-ab2d-40e9-9e51-f10f1aa7077c admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:40:59.866513 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:DELETE=6,keystone:SELECT=90,keystone:INSERT=4 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:40:59.913379 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c65acdca-c1a8-4cbd-a8e5-016b289aaad8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:40:59.914041 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c65acdca-c1a8-4cbd-a8e5-016b289aaad8 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:40:59.914316 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c65acdca-c1a8-4cbd-a8e5-016b289aaad8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:40:59.914585 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c65acdca-c1a8-4cbd-a8e5-016b289aaad8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:40:59.915233 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c65acdca-c1a8-4cbd-a8e5-016b289aaad8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:40:59.916823 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c65acdca-c1a8-4cbd-a8e5-016b289aaad8 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:40:59.928293 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1224/2456] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:40:59 2022] GET /identity/v3/roles => generated 1420 bytes in 98 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:40:59.932689 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d7b47cc-4019-4891-9ac4-432a402009e6 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:40:59.938529 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:40:59.947092 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d7b47cc-4019-4891-9ac4-432a402009e6 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:40:59.952334 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b6f9f24-5f30-4086-a30e-8729eb3e1238 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:40:59.953591 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b6f9f24-5f30-4086-a30e-8729eb3e1238 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:40:59.953936 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b6f9f24-5f30-4086-a30e-8729eb3e1238 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:40:59.954397 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b6f9f24-5f30-4086-a30e-8729eb3e1238 admin admin] PATH_INFO: `/v3/projects/355fe3e8b7754671aa90d66f1783ec01/users/66e0c0ed344440d1b93c961976feac67/roles/a6e562cdc96c4be691d638990f8403fd` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:40:59.956926 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b6f9f24-5f30-4086-a30e-8729eb3e1238 admin admin] RBAC: Authorizing `identity:create_grant(project_id=355fe3e8b7754671aa90d66f1783ec01, user_id=66e0c0ed344440d1b93c961976feac67, role_id=a6e562cdc96c4be691d638990f8403fd)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:40:59.960430 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b6f9f24-5f30-4086-a30e-8729eb3e1238 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:40:59.971903 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1233/2457] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:40:59 2022] PUT /identity/v3/projects/355fe3e8b7754671aa90d66f1783ec01/users/66e0c0ed344440d1b93c961976feac67/roles/a6e562cdc96c4be691d638990f8403fd => generated 0 bytes in 40 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 21:40:59.984823 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d628371d-a986-45bb-a2b6-f825f1236a50 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:40:59.985182 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d628371d-a986-45bb-a2b6-f825f1236a50 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:40:59.985466 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d628371d-a986-45bb-a2b6-f825f1236a50 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:00.017300 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-d628371d-a986-45bb-a2b6-f825f1236a50 None None] MFA Rules not processed for user `66e0c0ed344440d1b93c961976feac67`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:41:00.018638 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-d628371d-a986-45bb-a2b6-f825f1236a50 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:00.091266 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1225/2458] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:40:59 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:41:00.103330 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b6f9f24-5f30-4086-a30e-8729eb3e1238 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:00.105099 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-9b6f9f24-5f30-4086-a30e-8729eb3e1238 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:00.169643 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b6f9f24-5f30-4086-a30e-8729eb3e1238 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:00.173295 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76c6969b-f248-4b7f-8760-39c84692e7d6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dc226f7ddcb49ecacb51ce57fdfc11b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:00.173949 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c6969b-f248-4b7f-8760-39c84692e7d6 service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:00.174917 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c6969b-f248-4b7f-8760-39c84692e7d6 service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:00.175232 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c6969b-f248-4b7f-8760-39c84692e7d6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:00.185027 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76c6969b-f248-4b7f-8760-39c84692e7d6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:00.186527 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76c6969b-f248-4b7f-8760-39c84692e7d6 service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:00.198413 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1234/2459] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:41:00 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:41:00.297173 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d628371d-a986-45bb-a2b6-f825f1236a50 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:00.299098 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-d628371d-a986-45bb-a2b6-f825f1236a50 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:00.313123 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d628371d-a986-45bb-a2b6-f825f1236a50 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:00.316790 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90dc9767-d64d-4fc1-bf44-aad42c073a40 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dc226f7ddcb49ecacb51ce57fdfc11b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:00.317815 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90dc9767-d64d-4fc1-bf44-aad42c073a40 service cinder] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:00.317815 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90dc9767-d64d-4fc1-bf44-aad42c073a40 service cinder] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:00.318049 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90dc9767-d64d-4fc1-bf44-aad42c073a40 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:00.379977 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90dc9767-d64d-4fc1-bf44-aad42c073a40 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:00.381569 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90dc9767-d64d-4fc1-bf44-aad42c073a40 service cinder] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:00.390883 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1226/2460] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:41:00 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:41:00.707654 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76c6969b-f248-4b7f-8760-39c84692e7d6 service cinder] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:00.775782 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76c6969b-f248-4b7f-8760-39c84692e7d6 service cinder] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:00.781015 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cc106d0-af78-454a-b431-31629712d4f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:00.781706 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc106d0-af78-454a-b431-31629712d4f0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:00.782029 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc106d0-af78-454a-b431-31629712d4f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:00.782309 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc106d0-af78-454a-b431-31629712d4f0 admin admin] PATH_INFO: `/v3/users/ea135e9ede274721b13dfa01f033aa7f` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:00.784118 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cc106d0-af78-454a-b431-31629712d4f0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ea135e9ede274721b13dfa01f033aa7f)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:00.786078 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cc106d0-af78-454a-b431-31629712d4f0 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:00.846835 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-5cc106d0-af78-454a-b431-31629712d4f0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ea135e9ede274721b13dfa01f033aa7f'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:00.866127 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-5cc106d0-af78-454a-b431-31629712d4f0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ea135e9ede274721b13dfa01f033aa7f'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:00.866881 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-5cc106d0-af78-454a-b431-31629712d4f0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ea135e9ede274721b13dfa01f033aa7f'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:00.867510 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-5cc106d0-af78-454a-b431-31629712d4f0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ea135e9ede274721b13dfa01f033aa7f'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:00.875888 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-5cc106d0-af78-454a-b431-31629712d4f0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ea135e9ede274721b13dfa01f033aa7f'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:00.892560 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1235/2461] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:41:00 2022] DELETE /identity/v3/users/ea135e9ede274721b13dfa01f033aa7f => generated 0 bytes in 186 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:41:00.898517 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-545ea89e-41e7-4a16-aaf8-89de3514c423 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:00.899064 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-545ea89e-41e7-4a16-aaf8-89de3514c423 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:00.899189 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-545ea89e-41e7-4a16-aaf8-89de3514c423 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:00.928473 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-545ea89e-41e7-4a16-aaf8-89de3514c423 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:41:00.929716 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-545ea89e-41e7-4a16-aaf8-89de3514c423 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:00.975306 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:41:01.002033 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1227/2462] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:41:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 21:41:01.012663 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cc106d0-af78-454a-b431-31629712d4f0 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:01.014140 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-5cc106d0-af78-454a-b431-31629712d4f0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:01.028067 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:41:01.076360 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cc106d0-af78-454a-b431-31629712d4f0 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:01.080531 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daccc080-aa48-4a7f-808f-eb6431f00c74 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:01.081169 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daccc080-aa48-4a7f-808f-eb6431f00c74 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:01.081453 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daccc080-aa48-4a7f-808f-eb6431f00c74 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:01.081728 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daccc080-aa48-4a7f-808f-eb6431f00c74 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:01.091227 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daccc080-aa48-4a7f-808f-eb6431f00c74 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:01.092660 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daccc080-aa48-4a7f-808f-eb6431f00c74 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:01.103416 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1236/2463] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:41:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:41:01.512033 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-545ea89e-41e7-4a16-aaf8-89de3514c423 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:01.513843 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-545ea89e-41e7-4a16-aaf8-89de3514c423 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:01.534893 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-545ea89e-41e7-4a16-aaf8-89de3514c423 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:01.540202 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6837386c-d53d-42fe-be87-580e4be8d05c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:01.541118 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6837386c-d53d-42fe-be87-580e4be8d05c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:01.541523 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6837386c-d53d-42fe-be87-580e4be8d05c admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:01.541753 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6837386c-d53d-42fe-be87-580e4be8d05c admin admin] PATH_INFO: `/v3/projects/0eb22bb4364e4999b0b0a399ff81c969` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:01.543022 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6837386c-d53d-42fe-be87-580e4be8d05c admin admin] RBAC: Authorizing `identity:delete_project(project_id=0eb22bb4364e4999b0b0a399ff81c969)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:01.546443 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6837386c-d53d-42fe-be87-580e4be8d05c admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:01.608383 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-6837386c-d53d-42fe-be87-580e4be8d05c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0eb22bb4364e4999b0b0a399ff81c969', 'request_id': 'req-6837386c-d53d-42fe-be87-580e4be8d05c'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:01.614321 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-6837386c-d53d-42fe-be87-580e4be8d05c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0eb22bb4364e4999b0b0a399ff81c969', 'request_id': 'req-6837386c-d53d-42fe-be87-580e4be8d05c'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:01.620791 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-6837386c-d53d-42fe-be87-580e4be8d05c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0eb22bb4364e4999b0b0a399ff81c969', 'request_id': 'req-6837386c-d53d-42fe-be87-580e4be8d05c'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:01.634849 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-6837386c-d53d-42fe-be87-580e4be8d05c admin admin] The token cache is being invalidate because project 0eb22bb4364e4999b0b0a399ff81c969 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81434) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:41:01.635139 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-6837386c-d53d-42fe-be87-580e4be8d05c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:01.635915 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-6837386c-d53d-42fe-be87-580e4be8d05c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:01.637464 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1228/2464] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:41:01 2022] DELETE /identity/v3/projects/0eb22bb4364e4999b0b0a399ff81c969 => generated 0 bytes in 126 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:41:01.641705 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daccc080-aa48-4a7f-808f-eb6431f00c74 service neutron] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:01.652514 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daccc080-aa48-4a7f-808f-eb6431f00c74 service neutron] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:01.721638 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb058f96-63d0-4c69-9c5e-f5dc89887170 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:01.722385 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb058f96-63d0-4c69-9c5e-f5dc89887170 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:01.722663 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb058f96-63d0-4c69-9c5e-f5dc89887170 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:01.722941 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb058f96-63d0-4c69-9c5e-f5dc89887170 admin admin] PATH_INFO: `/v3/users/66e0c0ed344440d1b93c961976feac67` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:01.724103 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb058f96-63d0-4c69-9c5e-f5dc89887170 admin admin] RBAC: Authorizing `identity:delete_user(user_id=66e0c0ed344440d1b93c961976feac67)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:01.726145 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb058f96-63d0-4c69-9c5e-f5dc89887170 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:01.787364 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-eb058f96-63d0-4c69-9c5e-f5dc89887170 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '66e0c0ed344440d1b93c961976feac67'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:01.804364 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-eb058f96-63d0-4c69-9c5e-f5dc89887170 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '66e0c0ed344440d1b93c961976feac67'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:01.805152 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-eb058f96-63d0-4c69-9c5e-f5dc89887170 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '66e0c0ed344440d1b93c961976feac67'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:01.805953 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-eb058f96-63d0-4c69-9c5e-f5dc89887170 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '66e0c0ed344440d1b93c961976feac67'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:01.820479 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-eb058f96-63d0-4c69-9c5e-f5dc89887170 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '66e0c0ed344440d1b93c961976feac67'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:01.840069 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1237/2465] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:41:01 2022] DELETE /identity/v3/users/66e0c0ed344440d1b93c961976feac67 => generated 0 bytes in 199 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 21:41:02.220320 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6837386c-d53d-42fe-be87-580e4be8d05c admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:02.221931 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-6837386c-d53d-42fe-be87-580e4be8d05c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:02.235521 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6837386c-d53d-42fe-be87-580e4be8d05c admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:02.302870 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2e20131-ae97-4cd4-9e98-78e1046ef9b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:02.303513 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2e20131-ae97-4cd4-9e98-78e1046ef9b2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:02.303800 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2e20131-ae97-4cd4-9e98-78e1046ef9b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:02.304076 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2e20131-ae97-4cd4-9e98-78e1046ef9b2 admin admin] PATH_INFO: `/v3/projects/355fe3e8b7754671aa90d66f1783ec01` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:02.305479 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2e20131-ae97-4cd4-9e98-78e1046ef9b2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=355fe3e8b7754671aa90d66f1783ec01)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:02.307582 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2e20131-ae97-4cd4-9e98-78e1046ef9b2 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:02.347498 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9744e2a-9736-4094-96b9-60b66350c283 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:02.347994 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9744e2a-9736-4094-96b9-60b66350c283 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:02.348092 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9744e2a-9736-4094-96b9-60b66350c283 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:02.369205 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-f9744e2a-9736-4094-96b9-60b66350c283 None None] MFA Rules not processed for user `0c0b0da87b124de1be00faec4bbdb404`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:41:02.370484 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-f9744e2a-9736-4094-96b9-60b66350c283 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:02.373447 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-d2e20131-ae97-4cd4-9e98-78e1046ef9b2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '355fe3e8b7754671aa90d66f1783ec01', 'request_id': 'req-d2e20131-ae97-4cd4-9e98-78e1046ef9b2'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:02.381520 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-d2e20131-ae97-4cd4-9e98-78e1046ef9b2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '355fe3e8b7754671aa90d66f1783ec01', 'request_id': 'req-d2e20131-ae97-4cd4-9e98-78e1046ef9b2'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:02.388453 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-d2e20131-ae97-4cd4-9e98-78e1046ef9b2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '355fe3e8b7754671aa90d66f1783ec01', 'request_id': 'req-d2e20131-ae97-4cd4-9e98-78e1046ef9b2'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:02.398659 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:41:02.406022 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-d2e20131-ae97-4cd4-9e98-78e1046ef9b2 admin admin] The token cache is being invalidate because project 355fe3e8b7754671aa90d66f1783ec01 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81434) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:41:02.406655 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-d2e20131-ae97-4cd4-9e98-78e1046ef9b2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:02.408192 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-d2e20131-ae97-4cd4-9e98-78e1046ef9b2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:02.411108 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1229/2466] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:41:02 2022] DELETE /identity/v3/projects/355fe3e8b7754671aa90d66f1783ec01 => generated 0 bytes in 191 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:41:02.414552 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:41:02.419231 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93b41f95-9cf1-4547-ac1a-d5759e897710 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:02.419571 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93b41f95-9cf1-4547-ac1a-d5759e897710 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:02.419792 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93b41f95-9cf1-4547-ac1a-d5759e897710 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:02.442456 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-93b41f95-9cf1-4547-ac1a-d5759e897710 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:41:02.443585 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-93b41f95-9cf1-4547-ac1a-d5759e897710 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:02.447616 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1238/2467] 10.222.0.90 () {58 vars in 1009 bytes} [Mon Aug 29 21:41:02 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 21:41:02.460588 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9744e2a-9736-4094-96b9-60b66350c283 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:02.462130 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-f9744e2a-9736-4094-96b9-60b66350c283 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:02.513820 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1230/2468] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:41:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 21:41:02.522651 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f92f6aee-3d37-4d27-a631-062c238a2d34 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:02.522942 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f92f6aee-3d37-4d27-a631-062c238a2d34 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:02.523296 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f92f6aee-3d37-4d27-a631-062c238a2d34 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:02.541039 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9744e2a-9736-4094-96b9-60b66350c283 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:02.543229 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-f92f6aee-3d37-4d27-a631-062c238a2d34 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:41:02.544498 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-f92f6aee-3d37-4d27-a631-062c238a2d34 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:02.546712 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d46a58d0-967c-4f69-9c31-af347badf044 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '139bbbde6d884b42b5440727926e2c58', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:02.547337 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d46a58d0-967c-4f69-9c31-af347badf044 service swift] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:02.547623 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d46a58d0-967c-4f69-9c31-af347badf044 service swift] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:02.547899 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d46a58d0-967c-4f69-9c31-af347badf044 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:02.560167 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1231/2469] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:41:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 21:41:02.563653 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f92f6aee-3d37-4d27-a631-062c238a2d34 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:02.563847 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d46a58d0-967c-4f69-9c31-af347badf044 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:02.567800 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d46a58d0-967c-4f69-9c31-af347badf044 service swift] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:02.575073 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f92f6aee-3d37-4d27-a631-062c238a2d34 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:02.580218 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2892770-df64-4f74-a316-81050fd2b455 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:02.581918 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2892770-df64-4f74-a316-81050fd2b455 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:02.582206 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2892770-df64-4f74-a316-81050fd2b455 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:02.582482 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2892770-df64-4f74-a316-81050fd2b455 admin admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:02.583138 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2892770-df64-4f74-a316-81050fd2b455 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:41:02.583618 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2892770-df64-4f74-a316-81050fd2b455 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:02.585081 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2892770-df64-4f74-a316-81050fd2b455 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:02.585595 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1239/2470] 10.222.0.90 () {60 vars in 1384 bytes} [Mon Aug 29 21:41:02 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 125 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Aug 29 21:41:02.600485 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1232/2471] 10.222.0.90 () {58 vars in 1153 bytes} [Mon Aug 29 21:41:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:41:02.605057 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d46a58d0-967c-4f69-9c31-af347badf044 service swift] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:02.617865 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d46a58d0-967c-4f69-9c31-af347badf044 service swift] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:02.623095 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-526a57fc-1187-4246-8050-352a641300e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:02.623780 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-526a57fc-1187-4246-8050-352a641300e0 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:02.624043 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-526a57fc-1187-4246-8050-352a641300e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:02.624306 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-526a57fc-1187-4246-8050-352a641300e0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:02.624936 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-526a57fc-1187-4246-8050-352a641300e0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:02.626974 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-526a57fc-1187-4246-8050-352a641300e0 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:02.718520 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1240/2472] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:41:02 2022] POST /identity/v3/projects => generated 377 bytes in 114 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 21:41:02.723262 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2892770-df64-4f74-a316-81050fd2b455 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:02.757970 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2892770-df64-4f74-a316-81050fd2b455 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:02.957758 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41c6dc93-39e6-4939-b2dd-3bcb2d0643af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:02.958466 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41c6dc93-39e6-4939-b2dd-3bcb2d0643af admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:02.958753 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41c6dc93-39e6-4939-b2dd-3bcb2d0643af admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:02.959037 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41c6dc93-39e6-4939-b2dd-3bcb2d0643af admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:02.960603 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41c6dc93-39e6-4939-b2dd-3bcb2d0643af admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:02.963186 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41c6dc93-39e6-4939-b2dd-3bcb2d0643af admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:03.090395 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1233/2473] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:41:02 2022] POST /identity/v3/users => generated 289 bytes in 368 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:41:03.101490 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-526a57fc-1187-4246-8050-352a641300e0 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:03.123716 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-526a57fc-1187-4246-8050-352a641300e0 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:03.132087 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-708daa3a-21e4-4588-985e-79d2ce35e637 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:03.132964 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-708daa3a-21e4-4588-985e-79d2ce35e637 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:03.133421 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-708daa3a-21e4-4588-985e-79d2ce35e637 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:03.134038 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-708daa3a-21e4-4588-985e-79d2ce35e637 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:03.134883 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-708daa3a-21e4-4588-985e-79d2ce35e637 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:03.136916 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-708daa3a-21e4-4588-985e-79d2ce35e637 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:03.157923 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41c6dc93-39e6-4939-b2dd-3bcb2d0643af admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:03.173630 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1241/2474] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:41:03 2022] GET /identity/v3/roles => generated 1420 bytes in 72 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:41:03.175535 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41c6dc93-39e6-4939-b2dd-3bcb2d0643af admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:03.182843 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f3c07a9-c27f-461b-b402-d16909feb83b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:03.185077 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f3c07a9-c27f-461b-b402-d16909feb83b admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:03.185772 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f3c07a9-c27f-461b-b402-d16909feb83b admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:03.186304 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f3c07a9-c27f-461b-b402-d16909feb83b admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:03.187130 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f3c07a9-c27f-461b-b402-d16909feb83b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:03.189244 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f3c07a9-c27f-461b-b402-d16909feb83b admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:03.212221 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1234/2475] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:41:03 2022] GET /identity/v3/roles => generated 1420 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:41:03.217967 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-708daa3a-21e4-4588-985e-79d2ce35e637 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:03.253439 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-708daa3a-21e4-4588-985e-79d2ce35e637 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:03.272330 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cf7fce9-e12f-41c3-8384-6a9f708ae90e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:03.272330 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cf7fce9-e12f-41c3-8384-6a9f708ae90e admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:03.272688 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cf7fce9-e12f-41c3-8384-6a9f708ae90e admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:03.272932 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cf7fce9-e12f-41c3-8384-6a9f708ae90e admin admin] PATH_INFO: `/v3/projects/897f2055d8e14a58b2c85a47e64a6601/users/af5249fcd973480ab998de75c4b19647/roles/02e242d5520740d4b77a5f4008c7eb57` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:03.329493 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cf7fce9-e12f-41c3-8384-6a9f708ae90e admin admin] RBAC: Authorizing `identity:create_grant(project_id=897f2055d8e14a58b2c85a47e64a6601, user_id=af5249fcd973480ab998de75c4b19647, role_id=02e242d5520740d4b77a5f4008c7eb57)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:03.334528 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cf7fce9-e12f-41c3-8384-6a9f708ae90e admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:03.374466 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1242/2476] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:41:03 2022] PUT /identity/v3/projects/897f2055d8e14a58b2c85a47e64a6601/users/af5249fcd973480ab998de75c4b19647/roles/02e242d5520740d4b77a5f4008c7eb57 => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:41:03.388629 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c91f233-f0f3-4436-a67d-e5488d5fe9e7 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:03.389339 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c91f233-f0f3-4436-a67d-e5488d5fe9e7 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:03.389898 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c91f233-f0f3-4436-a67d-e5488d5fe9e7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:03.467178 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-2c91f233-f0f3-4436-a67d-e5488d5fe9e7 None None] MFA Rules not processed for user `af5249fcd973480ab998de75c4b19647`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:41:03.468692 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-2c91f233-f0f3-4436-a67d-e5488d5fe9e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:03.594831 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:41:03.626694 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1235/2477] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:41:03 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 240 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:41:03.641859 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cf7fce9-e12f-41c3-8384-6a9f708ae90e admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:03.644437 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-5cf7fce9-e12f-41c3-8384-6a9f708ae90e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:03.749980 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cf7fce9-e12f-41c3-8384-6a9f708ae90e admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:03.768720 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a14f3e72-d51f-41cc-8795-eab26f280a00 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c63a4eaf48ad43529f0aa328f2ebfd29', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:03.769655 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a14f3e72-d51f-41cc-8795-eab26f280a00 service glance] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:03.770236 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a14f3e72-d51f-41cc-8795-eab26f280a00 service glance] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:03.770732 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a14f3e72-d51f-41cc-8795-eab26f280a00 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:03.785021 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a14f3e72-d51f-41cc-8795-eab26f280a00 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:03.786849 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a14f3e72-d51f-41cc-8795-eab26f280a00 service glance] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:03.802156 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1243/2478] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:41:03 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 159 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 21:41:11.273314 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c91f233-f0f3-4436-a67d-e5488d5fe9e7 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:11.275181 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-2c91f233-f0f3-4436-a67d-e5488d5fe9e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:11.371484 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c91f233-f0f3-4436-a67d-e5488d5fe9e7 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:11.381919 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5437b3a8-408c-4a5b-a402-663f6c39a75e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:11.382594 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5437b3a8-408c-4a5b-a402-663f6c39a75e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:11.382878 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5437b3a8-408c-4a5b-a402-663f6c39a75e admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:11.383180 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5437b3a8-408c-4a5b-a402-663f6c39a75e admin admin] PATH_INFO: `/v3/users/2cc77a1abc0d45a38ef0bdc358e0e9ce` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:11.384409 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5437b3a8-408c-4a5b-a402-663f6c39a75e admin admin] RBAC: Authorizing `identity:delete_user(user_id=2cc77a1abc0d45a38ef0bdc358e0e9ce)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:11.386740 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5437b3a8-408c-4a5b-a402-663f6c39a75e admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:11.458383 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-5437b3a8-408c-4a5b-a402-663f6c39a75e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2cc77a1abc0d45a38ef0bdc358e0e9ce'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:11.473706 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-5437b3a8-408c-4a5b-a402-663f6c39a75e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2cc77a1abc0d45a38ef0bdc358e0e9ce'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:11.485111 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-5437b3a8-408c-4a5b-a402-663f6c39a75e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2cc77a1abc0d45a38ef0bdc358e0e9ce'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:11.485966 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-5437b3a8-408c-4a5b-a402-663f6c39a75e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2cc77a1abc0d45a38ef0bdc358e0e9ce'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:11.486699 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-5437b3a8-408c-4a5b-a402-663f6c39a75e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2cc77a1abc0d45a38ef0bdc358e0e9ce'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:11.514700 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1236/2479] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:41:11 2022] DELETE /identity/v3/users/2cc77a1abc0d45a38ef0bdc358e0e9ce => generated 0 bytes in 242 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:41:11.520746 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a3e0dea-27e4-42e8-8fa6-baa9680ce692 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:11.521307 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a3e0dea-27e4-42e8-8fa6-baa9680ce692 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:11.521586 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a3e0dea-27e4-42e8-8fa6-baa9680ce692 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:11.542528 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-8a3e0dea-27e4-42e8-8fa6-baa9680ce692 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:41:11.543761 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-8a3e0dea-27e4-42e8-8fa6-baa9680ce692 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:11.556377 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:41:11.633751 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1244/2480] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:41:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 21:41:11.646601 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5437b3a8-408c-4a5b-a402-663f6c39a75e admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:11.648088 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-5437b3a8-408c-4a5b-a402-663f6c39a75e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:11.737108 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5437b3a8-408c-4a5b-a402-663f6c39a75e admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:11.741477 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76034dcc-02a5-4a85-8bca-588bbd774cdf service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:11.742149 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76034dcc-02a5-4a85-8bca-588bbd774cdf service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:11.742435 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76034dcc-02a5-4a85-8bca-588bbd774cdf service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:11.742712 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76034dcc-02a5-4a85-8bca-588bbd774cdf service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:11.753536 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76034dcc-02a5-4a85-8bca-588bbd774cdf service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:11.755606 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76034dcc-02a5-4a85-8bca-588bbd774cdf service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:11.768103 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1237/2481] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:41:11 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 123 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:41:12.139190 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a3e0dea-27e4-42e8-8fa6-baa9680ce692 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:12.140772 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-8a3e0dea-27e4-42e8-8fa6-baa9680ce692 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:12.154492 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a3e0dea-27e4-42e8-8fa6-baa9680ce692 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:12.160107 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fde9340a-1af4-4ab3-9de5-c94d519429d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:12.160860 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fde9340a-1af4-4ab3-9de5-c94d519429d1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:12.161209 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fde9340a-1af4-4ab3-9de5-c94d519429d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:12.161579 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fde9340a-1af4-4ab3-9de5-c94d519429d1 admin admin] PATH_INFO: `/v3/projects/0271b2ec720a402da3fcaad65ce4f7fd` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:12.162886 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fde9340a-1af4-4ab3-9de5-c94d519429d1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0271b2ec720a402da3fcaad65ce4f7fd)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:12.165012 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fde9340a-1af4-4ab3-9de5-c94d519429d1 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:12.234636 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-fde9340a-1af4-4ab3-9de5-c94d519429d1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0271b2ec720a402da3fcaad65ce4f7fd', 'request_id': 'req-fde9340a-1af4-4ab3-9de5-c94d519429d1'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:12.241745 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-fde9340a-1af4-4ab3-9de5-c94d519429d1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0271b2ec720a402da3fcaad65ce4f7fd', 'request_id': 'req-fde9340a-1af4-4ab3-9de5-c94d519429d1'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:12.247860 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-fde9340a-1af4-4ab3-9de5-c94d519429d1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0271b2ec720a402da3fcaad65ce4f7fd', 'request_id': 'req-fde9340a-1af4-4ab3-9de5-c94d519429d1'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:12.263628 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-fde9340a-1af4-4ab3-9de5-c94d519429d1 admin admin] The token cache is being invalidate because project 0271b2ec720a402da3fcaad65ce4f7fd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:41:12.263988 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-fde9340a-1af4-4ab3-9de5-c94d519429d1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:12.264573 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-fde9340a-1af4-4ab3-9de5-c94d519429d1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:12.266120 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1245/2482] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:41:12 2022] DELETE /identity/v3/projects/0271b2ec720a402da3fcaad65ce4f7fd => generated 0 bytes in 127 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 21:41:12.270678 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76034dcc-02a5-4a85-8bca-588bbd774cdf service neutron] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:12.281196 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76034dcc-02a5-4a85-8bca-588bbd774cdf service neutron] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:12.292668 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:41:12.354964 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d710e70-7de2-459e-b79f-620b9414c9c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:12.355607 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d710e70-7de2-459e-b79f-620b9414c9c6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:12.355887 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d710e70-7de2-459e-b79f-620b9414c9c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:12.356231 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d710e70-7de2-459e-b79f-620b9414c9c6 admin admin] PATH_INFO: `/v3/users/a82964d4cd6e4872b99b5aac12b77ca1` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:12.357425 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d710e70-7de2-459e-b79f-620b9414c9c6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a82964d4cd6e4872b99b5aac12b77ca1)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:12.359428 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d710e70-7de2-459e-b79f-620b9414c9c6 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:12.413609 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-2d710e70-7de2-459e-b79f-620b9414c9c6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a82964d4cd6e4872b99b5aac12b77ca1'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:12.421667 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-2d710e70-7de2-459e-b79f-620b9414c9c6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a82964d4cd6e4872b99b5aac12b77ca1'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:12.431695 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-2d710e70-7de2-459e-b79f-620b9414c9c6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a82964d4cd6e4872b99b5aac12b77ca1'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:12.432635 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-2d710e70-7de2-459e-b79f-620b9414c9c6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a82964d4cd6e4872b99b5aac12b77ca1'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:12.433306 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-2d710e70-7de2-459e-b79f-620b9414c9c6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a82964d4cd6e4872b99b5aac12b77ca1'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:12.451169 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1238/2483] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:41:12 2022] DELETE /identity/v3/users/a82964d4cd6e4872b99b5aac12b77ca1 => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:41:12.757103 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fde9340a-1af4-4ab3-9de5-c94d519429d1 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:12.758862 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-fde9340a-1af4-4ab3-9de5-c94d519429d1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:12.774193 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fde9340a-1af4-4ab3-9de5-c94d519429d1 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:12.844190 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60bc27f6-3e07-4191-be1a-1c7115dbebaf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:12.845091 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60bc27f6-3e07-4191-be1a-1c7115dbebaf admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:12.845454 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60bc27f6-3e07-4191-be1a-1c7115dbebaf admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:12.845964 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60bc27f6-3e07-4191-be1a-1c7115dbebaf admin admin] PATH_INFO: `/v3/projects/16dc7391579b41edaf0988b56b6b53f8` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:12.847278 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60bc27f6-3e07-4191-be1a-1c7115dbebaf admin admin] RBAC: Authorizing `identity:delete_project(project_id=16dc7391579b41edaf0988b56b6b53f8)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:12.849394 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60bc27f6-3e07-4191-be1a-1c7115dbebaf admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:12.904301 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:41:12.917805 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-60bc27f6-3e07-4191-be1a-1c7115dbebaf admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '16dc7391579b41edaf0988b56b6b53f8', 'request_id': 'req-60bc27f6-3e07-4191-be1a-1c7115dbebaf'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:12.929776 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-60bc27f6-3e07-4191-be1a-1c7115dbebaf admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '16dc7391579b41edaf0988b56b6b53f8', 'request_id': 'req-60bc27f6-3e07-4191-be1a-1c7115dbebaf'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:12.936413 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-60bc27f6-3e07-4191-be1a-1c7115dbebaf admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '16dc7391579b41edaf0988b56b6b53f8', 'request_id': 'req-60bc27f6-3e07-4191-be1a-1c7115dbebaf'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:12.950598 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-60bc27f6-3e07-4191-be1a-1c7115dbebaf admin admin] The token cache is being invalidate because project 16dc7391579b41edaf0988b56b6b53f8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:41:12.951130 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-60bc27f6-3e07-4191-be1a-1c7115dbebaf admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:12.951691 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-60bc27f6-3e07-4191-be1a-1c7115dbebaf admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:12.953563 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1246/2484] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:41:12 2022] DELETE /identity/v3/projects/16dc7391579b41edaf0988b56b6b53f8 => generated 0 bytes in 197 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:41:12.958926 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-572a1654-2c49-4986-b5f5-657a2902fe7e None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:12.959396 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-572a1654-2c49-4986-b5f5-657a2902fe7e None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:12.959509 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-572a1654-2c49-4986-b5f5-657a2902fe7e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:12.979551 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-572a1654-2c49-4986-b5f5-657a2902fe7e None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:41:12.980780 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-572a1654-2c49-4986-b5f5-657a2902fe7e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:13.043884 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1239/2485] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:41:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:41:13.054264 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb8b02e1-53cd-4f40-9171-856391068e49 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:13.054623 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb8b02e1-53cd-4f40-9171-856391068e49 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:13.054830 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb8b02e1-53cd-4f40-9171-856391068e49 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:13.075951 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-cb8b02e1-53cd-4f40-9171-856391068e49 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:41:13.077374 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-cb8b02e1-53cd-4f40-9171-856391068e49 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:13.086307 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1247/2486] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:41:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:41:13.090361 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-572a1654-2c49-4986-b5f5-657a2902fe7e None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:13.102001 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-572a1654-2c49-4986-b5f5-657a2902fe7e None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:13.106990 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dee422ee-cbfc-4843-b0f7-2af3bdb89a13 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:13.107687 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee422ee-cbfc-4843-b0f7-2af3bdb89a13 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:13.108220 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee422ee-cbfc-4843-b0f7-2af3bdb89a13 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:13.108565 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee422ee-cbfc-4843-b0f7-2af3bdb89a13 admin admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:13.109295 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dee422ee-cbfc-4843-b0f7-2af3bdb89a13 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:41:13.109673 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dee422ee-cbfc-4843-b0f7-2af3bdb89a13 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:13.112302 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dee422ee-cbfc-4843-b0f7-2af3bdb89a13 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:13.125356 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1240/2487] 10.222.0.90 () {58 vars in 1153 bytes} [Mon Aug 29 21:41:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:41:13.130026 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb8b02e1-53cd-4f40-9171-856391068e49 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:13.147586 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb8b02e1-53cd-4f40-9171-856391068e49 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:13.153472 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f8944ec-b732-4d31-9227-3e0377b96e3d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:13.154338 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f8944ec-b732-4d31-9227-3e0377b96e3d admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:13.154674 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f8944ec-b732-4d31-9227-3e0377b96e3d admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:13.154977 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f8944ec-b732-4d31-9227-3e0377b96e3d admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:13.155684 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f8944ec-b732-4d31-9227-3e0377b96e3d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:13.157568 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f8944ec-b732-4d31-9227-3e0377b96e3d admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:13.196876 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1248/2488] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:41:13 2022] POST /identity/v3/projects => generated 393 bytes in 64 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:41:13.198578 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dee422ee-cbfc-4843-b0f7-2af3bdb89a13 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:13.214008 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dee422ee-cbfc-4843-b0f7-2af3bdb89a13 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:13.232983 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:41:13.291539 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9de9b88b-d871-4dc3-9f54-c907484dd561 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:13.292641 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9de9b88b-d871-4dc3-9f54-c907484dd561 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:13.292641 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9de9b88b-d871-4dc3-9f54-c907484dd561 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:13.292873 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9de9b88b-d871-4dc3-9f54-c907484dd561 admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:13.293722 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9de9b88b-d871-4dc3-9f54-c907484dd561 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:13.295603 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9de9b88b-d871-4dc3-9f54-c907484dd561 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:13.325036 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1241/2489] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:41:13 2022] POST /identity/v3/users => generated 297 bytes in 127 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:41:13.329357 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f8944ec-b732-4d31-9227-3e0377b96e3d admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:13.340880 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f8944ec-b732-4d31-9227-3e0377b96e3d admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:13.346771 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2187acc5-a972-4b2d-9c8a-8be7d6ee15d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:13.347389 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2187acc5-a972-4b2d-9c8a-8be7d6ee15d3 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:13.347673 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2187acc5-a972-4b2d-9c8a-8be7d6ee15d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:13.347951 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2187acc5-a972-4b2d-9c8a-8be7d6ee15d3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:13.348602 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2187acc5-a972-4b2d-9c8a-8be7d6ee15d3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:13.350250 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2187acc5-a972-4b2d-9c8a-8be7d6ee15d3 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:13.362611 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1249/2490] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:41:13 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:41:13.366646 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9de9b88b-d871-4dc3-9f54-c907484dd561 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:13.379995 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9de9b88b-d871-4dc3-9f54-c907484dd561 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:13.384889 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2cc9987-2d2d-4ba2-9a21-10a5d55f6abd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:13.385509 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2cc9987-2d2d-4ba2-9a21-10a5d55f6abd admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:13.385809 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2cc9987-2d2d-4ba2-9a21-10a5d55f6abd admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:13.386094 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2cc9987-2d2d-4ba2-9a21-10a5d55f6abd admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:13.386984 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2cc9987-2d2d-4ba2-9a21-10a5d55f6abd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:13.388339 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2cc9987-2d2d-4ba2-9a21-10a5d55f6abd admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:13.400398 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1242/2491] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:41:13 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:41:13.404816 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2187acc5-a972-4b2d-9c8a-8be7d6ee15d3 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:13.418125 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2187acc5-a972-4b2d-9c8a-8be7d6ee15d3 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:13.425447 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-516323e2-3ba0-476c-b878-b97c960472f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:13.426403 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-516323e2-3ba0-476c-b878-b97c960472f1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:13.426697 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-516323e2-3ba0-476c-b878-b97c960472f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:13.426982 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-516323e2-3ba0-476c-b878-b97c960472f1 admin admin] PATH_INFO: `/v3/projects/d76043147ae248acb01fb0c9dab3c8cc/users/c4f1cc4131524f53b539782ec89c669a/roles/02e242d5520740d4b77a5f4008c7eb57` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:13.443539 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-516323e2-3ba0-476c-b878-b97c960472f1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d76043147ae248acb01fb0c9dab3c8cc, user_id=c4f1cc4131524f53b539782ec89c669a, role_id=02e242d5520740d4b77a5f4008c7eb57)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:13.446447 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-516323e2-3ba0-476c-b878-b97c960472f1 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:13.461850 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1250/2492] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:41:13 2022] PUT /identity/v3/projects/d76043147ae248acb01fb0c9dab3c8cc/users/c4f1cc4131524f53b539782ec89c669a/roles/02e242d5520740d4b77a5f4008c7eb57 => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:41:13.473360 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3162122-551e-4628-b2b0-fd15cfc9273d None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:13.473648 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3162122-551e-4628-b2b0-fd15cfc9273d None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:13.473972 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3162122-551e-4628-b2b0-fd15cfc9273d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:13.503933 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-c3162122-551e-4628-b2b0-fd15cfc9273d None None] MFA Rules not processed for user `c4f1cc4131524f53b539782ec89c669a`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:41:13.505175 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-c3162122-551e-4628-b2b0-fd15cfc9273d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:13.581279 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1243/2493] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:41:13 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:41:13.586204 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-516323e2-3ba0-476c-b878-b97c960472f1 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:13.657943 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-516323e2-3ba0-476c-b878-b97c960472f1 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:13.664823 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ec1506a-bfa4-4bd5-9c31-b013610c20ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:13.665451 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ec1506a-bfa4-4bd5-9c31-b013610c20ef admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:13.665737 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ec1506a-bfa4-4bd5-9c31-b013610c20ef admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:13.666047 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ec1506a-bfa4-4bd5-9c31-b013610c20ef admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:13.666714 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ec1506a-bfa4-4bd5-9c31-b013610c20ef admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:13.668664 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ec1506a-bfa4-4bd5-9c31-b013610c20ef admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:13.697227 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1251/2494] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:41:13 2022] POST /identity/v3/projects => generated 391 bytes in 112 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:41:13.701559 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3162122-551e-4628-b2b0-fd15cfc9273d None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:13.714290 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3162122-551e-4628-b2b0-fd15cfc9273d None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:13.790111 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-567b6abd-cc08-4439-8a0c-4229eae3d62a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:13.790736 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-567b6abd-cc08-4439-8a0c-4229eae3d62a admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:13.791150 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-567b6abd-cc08-4439-8a0c-4229eae3d62a admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:13.791457 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-567b6abd-cc08-4439-8a0c-4229eae3d62a admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:13.792108 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-567b6abd-cc08-4439-8a0c-4229eae3d62a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:13.794130 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-567b6abd-cc08-4439-8a0c-4229eae3d62a admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:13.817469 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1244/2495] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:41:13 2022] POST /identity/v3/users => generated 302 bytes in 116 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:41:13.821603 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ec1506a-bfa4-4bd5-9c31-b013610c20ef admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:13.833306 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ec1506a-bfa4-4bd5-9c31-b013610c20ef admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:13.838817 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ca5d8ff-0980-4a75-9165-b670332b5fc5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:13.839491 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca5d8ff-0980-4a75-9165-b670332b5fc5 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:13.839795 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca5d8ff-0980-4a75-9165-b670332b5fc5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:13.840071 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca5d8ff-0980-4a75-9165-b670332b5fc5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:13.840713 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ca5d8ff-0980-4a75-9165-b670332b5fc5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:13.842394 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ca5d8ff-0980-4a75-9165-b670332b5fc5 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:13.853641 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1252/2496] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:41:13 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:41:13.857851 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-567b6abd-cc08-4439-8a0c-4229eae3d62a admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:13.869052 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-567b6abd-cc08-4439-8a0c-4229eae3d62a admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:13.879252 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe2793c1-9bb8-464c-bc0c-28404ed73d68 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:13.879868 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe2793c1-9bb8-464c-bc0c-28404ed73d68 admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:13.880154 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe2793c1-9bb8-464c-bc0c-28404ed73d68 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:13.880436 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe2793c1-9bb8-464c-bc0c-28404ed73d68 admin admin] PATH_INFO: `/v3/projects/044c3d8848984ba6a3d829883a1de041/users/ea052aec744b48778ad23327985ed5a8/roles/a6e562cdc96c4be691d638990f8403fd` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:13.898686 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:41:13.903490 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe2793c1-9bb8-464c-bc0c-28404ed73d68 admin admin] RBAC: Authorizing `identity:create_grant(project_id=044c3d8848984ba6a3d829883a1de041, user_id=ea052aec744b48778ad23327985ed5a8, role_id=a6e562cdc96c4be691d638990f8403fd)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:13.906734 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe2793c1-9bb8-464c-bc0c-28404ed73d68 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:13.927519 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1245/2497] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:41:13 2022] PUT /identity/v3/projects/044c3d8848984ba6a3d829883a1de041/users/ea052aec744b48778ad23327985ed5a8/roles/a6e562cdc96c4be691d638990f8403fd => generated 0 bytes in 67 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:41:13.928796 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ca5d8ff-0980-4a75-9165-b670332b5fc5 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:13.940580 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ca5d8ff-0980-4a75-9165-b670332b5fc5 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:13.978439 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=92,keystone:DELETE=4,keystone:INSERT=4 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:41:14.006613 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efc89c00-9c87-4cee-a2cf-befdacb7fdde admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:14.007310 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc89c00-9c87-4cee-a2cf-befdacb7fdde admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:14.007533 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc89c00-9c87-4cee-a2cf-befdacb7fdde admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:14.007806 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc89c00-9c87-4cee-a2cf-befdacb7fdde admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:14.008463 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efc89c00-9c87-4cee-a2cf-befdacb7fdde admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:14.010149 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efc89c00-9c87-4cee-a2cf-befdacb7fdde admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:14.024768 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1253/2498] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:41:13 2022] GET /identity/v3/roles => generated 1420 bytes in 96 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:41:14.029166 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe2793c1-9bb8-464c-bc0c-28404ed73d68 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:14.040858 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe2793c1-9bb8-464c-bc0c-28404ed73d68 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:14.045662 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc6ffd7f-208d-4cd6-b7db-fe2117635642 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:14.046367 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc6ffd7f-208d-4cd6-b7db-fe2117635642 admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:14.046673 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc6ffd7f-208d-4cd6-b7db-fe2117635642 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:14.046925 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc6ffd7f-208d-4cd6-b7db-fe2117635642 admin admin] PATH_INFO: `/v3/projects/044c3d8848984ba6a3d829883a1de041/users/ea052aec744b48778ad23327985ed5a8/roles/a6e562cdc96c4be691d638990f8403fd` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:14.049535 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc6ffd7f-208d-4cd6-b7db-fe2117635642 admin admin] RBAC: Authorizing `identity:create_grant(project_id=044c3d8848984ba6a3d829883a1de041, user_id=ea052aec744b48778ad23327985ed5a8, role_id=a6e562cdc96c4be691d638990f8403fd)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:14.052201 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc6ffd7f-208d-4cd6-b7db-fe2117635642 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:14.062957 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1246/2499] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:41:14 2022] PUT /identity/v3/projects/044c3d8848984ba6a3d829883a1de041/users/ea052aec744b48778ad23327985ed5a8/roles/a6e562cdc96c4be691d638990f8403fd => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:41:14.075385 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e865736-5449-492a-819f-b216b082709c None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:14.075697 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e865736-5449-492a-819f-b216b082709c None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:14.075997 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e865736-5449-492a-819f-b216b082709c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:14.105551 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-6e865736-5449-492a-819f-b216b082709c None None] MFA Rules not processed for user `ea052aec744b48778ad23327985ed5a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:41:14.106879 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-6e865736-5449-492a-819f-b216b082709c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:14.186097 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1254/2500] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:41:14 2022] POST /identity/v3/auth/tokens => generated 3335 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 21:41:14.197921 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc6ffd7f-208d-4cd6-b7db-fe2117635642 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:14.199654 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-bc6ffd7f-208d-4cd6-b7db-fe2117635642 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:14.272382 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc6ffd7f-208d-4cd6-b7db-fe2117635642 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:14.276455 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d33bb893-099d-42d0-81b1-a7b15ba8b32a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dc226f7ddcb49ecacb51ce57fdfc11b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:14.277100 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d33bb893-099d-42d0-81b1-a7b15ba8b32a service cinder] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:14.277385 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d33bb893-099d-42d0-81b1-a7b15ba8b32a service cinder] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:14.277903 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d33bb893-099d-42d0-81b1-a7b15ba8b32a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:14.289472 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d33bb893-099d-42d0-81b1-a7b15ba8b32a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:14.291101 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d33bb893-099d-42d0-81b1-a7b15ba8b32a service cinder] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:14.302729 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1247/2501] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:41:14 2022] GET /identity/v3/auth/tokens => generated 3335 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:41:14.359683 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e865736-5449-492a-819f-b216b082709c None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:14.378342 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e865736-5449-492a-819f-b216b082709c None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:14.385442 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91330e00-aac6-48ba-8c7f-8187490f062d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dc226f7ddcb49ecacb51ce57fdfc11b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:14.386988 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91330e00-aac6-48ba-8c7f-8187490f062d service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:14.387497 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91330e00-aac6-48ba-8c7f-8187490f062d service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:14.388182 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91330e00-aac6-48ba-8c7f-8187490f062d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:14.490123 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91330e00-aac6-48ba-8c7f-8187490f062d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:14.490123 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91330e00-aac6-48ba-8c7f-8187490f062d service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:14.493289 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1255/2502] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:41:14 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 134 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:41:18.573719 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d33bb893-099d-42d0-81b1-a7b15ba8b32a service cinder] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:18.654351 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d33bb893-099d-42d0-81b1-a7b15ba8b32a service cinder] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:18.659866 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf9436b3-b50a-4109-9528-a7c889a274be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:18.660518 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf9436b3-b50a-4109-9528-a7c889a274be admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:18.660808 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf9436b3-b50a-4109-9528-a7c889a274be admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:18.661082 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf9436b3-b50a-4109-9528-a7c889a274be admin admin] PATH_INFO: `/v3/users/c4f1cc4131524f53b539782ec89c669a` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:18.662982 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf9436b3-b50a-4109-9528-a7c889a274be admin admin] RBAC: Authorizing `identity:delete_user(user_id=c4f1cc4131524f53b539782ec89c669a)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:18.664907 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf9436b3-b50a-4109-9528-a7c889a274be admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:18.729008 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-bf9436b3-b50a-4109-9528-a7c889a274be admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c4f1cc4131524f53b539782ec89c669a'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:18.737775 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-bf9436b3-b50a-4109-9528-a7c889a274be admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c4f1cc4131524f53b539782ec89c669a'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:18.750850 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-bf9436b3-b50a-4109-9528-a7c889a274be admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c4f1cc4131524f53b539782ec89c669a'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:18.752172 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-bf9436b3-b50a-4109-9528-a7c889a274be admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c4f1cc4131524f53b539782ec89c669a'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:18.753356 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-bf9436b3-b50a-4109-9528-a7c889a274be admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c4f1cc4131524f53b539782ec89c669a'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:18.775817 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=87,keystone:INSERT=3,keystone:DELETE=10 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:41:18.781377 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1248/2503] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:41:18 2022] DELETE /identity/v3/users/c4f1cc4131524f53b539782ec89c669a => generated 0 bytes in 208 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:41:18.787362 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f86d8e2e-fc7f-448c-b9ec-b061eb4a8050 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:18.787362 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f86d8e2e-fc7f-448c-b9ec-b061eb4a8050 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:18.787362 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f86d8e2e-fc7f-448c-b9ec-b061eb4a8050 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:18.810397 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-f86d8e2e-fc7f-448c-b9ec-b061eb4a8050 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:41:18.811668 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-f86d8e2e-fc7f-448c-b9ec-b061eb4a8050 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:18.880404 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:41:18.898747 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1256/2504] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:41:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 21:41:18.906818 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf9436b3-b50a-4109-9528-a7c889a274be admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:18.909776 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-bf9436b3-b50a-4109-9528-a7c889a274be admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:18.984755 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf9436b3-b50a-4109-9528-a7c889a274be admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:18.989763 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14af1aab-682c-471f-8901-7581448928d1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:18.990484 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14af1aab-682c-471f-8901-7581448928d1 service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:18.990955 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14af1aab-682c-471f-8901-7581448928d1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:18.991236 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14af1aab-682c-471f-8901-7581448928d1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:19.002802 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14af1aab-682c-471f-8901-7581448928d1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:19.004281 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14af1aab-682c-471f-8901-7581448928d1 service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:19.017025 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1249/2505] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:41:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 111 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:41:19.098056 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85c84619-6075-4cef-89b2-a58ec49db0fc None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:19.098912 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85c84619-6075-4cef-89b2-a58ec49db0fc None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:19.099758 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85c84619-6075-4cef-89b2-a58ec49db0fc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:19.124027 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-85c84619-6075-4cef-89b2-a58ec49db0fc None None] MFA Rules not processed for user `0c0b0da87b124de1be00faec4bbdb404`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:41:19.125363 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-85c84619-6075-4cef-89b2-a58ec49db0fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:19.212673 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1257/2506] 10.222.0.90 () {58 vars in 967 bytes} [Mon Aug 29 21:41:19 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:41:19.227655 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14af1aab-682c-471f-8901-7581448928d1 service neutron] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:19.229274 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-14af1aab-682c-471f-8901-7581448928d1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:19.317023 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14af1aab-682c-471f-8901-7581448928d1 service neutron] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:19.323229 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d11b015e-89b8-48d2-8cd8-415709f32798 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '139bbbde6d884b42b5440727926e2c58', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:19.323893 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d11b015e-89b8-48d2-8cd8-415709f32798 service swift] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:19.324187 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d11b015e-89b8-48d2-8cd8-415709f32798 service swift] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:19.324610 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d11b015e-89b8-48d2-8cd8-415709f32798 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:19.335160 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d11b015e-89b8-48d2-8cd8-415709f32798 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:19.336928 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d11b015e-89b8-48d2-8cd8-415709f32798 service swift] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:19.339889 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85c84619-6075-4cef-89b2-a58ec49db0fc None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:19.341297 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-85c84619-6075-4cef-89b2-a58ec49db0fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:19.350781 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1250/2507] 10.222.0.90 () {60 vars in 1384 bytes} [Mon Aug 29 21:41:19 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 121 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 29 21:41:19.363476 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85c84619-6075-4cef-89b2-a58ec49db0fc None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:19.372387 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-662c6ea7-b9c4-46d8-8f0f-bc21885295a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:19.373578 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-662c6ea7-b9c4-46d8-8f0f-bc21885295a1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:19.374138 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-662c6ea7-b9c4-46d8-8f0f-bc21885295a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:19.374614 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-662c6ea7-b9c4-46d8-8f0f-bc21885295a1 admin admin] PATH_INFO: `/v3/projects/d76043147ae248acb01fb0c9dab3c8cc` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:19.377443 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-662c6ea7-b9c4-46d8-8f0f-bc21885295a1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d76043147ae248acb01fb0c9dab3c8cc)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:19.380812 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-662c6ea7-b9c4-46d8-8f0f-bc21885295a1 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:19.464120 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-662c6ea7-b9c4-46d8-8f0f-bc21885295a1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd76043147ae248acb01fb0c9dab3c8cc', 'request_id': 'req-662c6ea7-b9c4-46d8-8f0f-bc21885295a1'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:19.472416 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-662c6ea7-b9c4-46d8-8f0f-bc21885295a1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd76043147ae248acb01fb0c9dab3c8cc', 'request_id': 'req-662c6ea7-b9c4-46d8-8f0f-bc21885295a1'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:19.480140 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-662c6ea7-b9c4-46d8-8f0f-bc21885295a1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd76043147ae248acb01fb0c9dab3c8cc', 'request_id': 'req-662c6ea7-b9c4-46d8-8f0f-bc21885295a1'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:19.506416 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-662c6ea7-b9c4-46d8-8f0f-bc21885295a1 admin admin] The token cache is being invalidate because project d76043147ae248acb01fb0c9dab3c8cc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:41:19.506940 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-662c6ea7-b9c4-46d8-8f0f-bc21885295a1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:19.507821 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-662c6ea7-b9c4-46d8-8f0f-bc21885295a1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:19.515087 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1258/2508] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:41:19 2022] DELETE /identity/v3/projects/d76043147ae248acb01fb0c9dab3c8cc => generated 0 bytes in 171 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:41:19.515998 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d11b015e-89b8-48d2-8cd8-415709f32798 service swift] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:19.528438 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d11b015e-89b8-48d2-8cd8-415709f32798 service swift] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:19.637639 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3fe7228-afa0-43d9-aabb-ba3c11a9f758 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:19.639048 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3fe7228-afa0-43d9-aabb-ba3c11a9f758 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:19.639669 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3fe7228-afa0-43d9-aabb-ba3c11a9f758 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:19.640390 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3fe7228-afa0-43d9-aabb-ba3c11a9f758 admin admin] PATH_INFO: `/v3/users/ea052aec744b48778ad23327985ed5a8` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:19.642119 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3fe7228-afa0-43d9-aabb-ba3c11a9f758 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ea052aec744b48778ad23327985ed5a8)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:19.644616 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3fe7228-afa0-43d9-aabb-ba3c11a9f758 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:19.709320 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:41:19.725745 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-f3fe7228-afa0-43d9-aabb-ba3c11a9f758 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ea052aec744b48778ad23327985ed5a8'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:19.733129 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-f3fe7228-afa0-43d9-aabb-ba3c11a9f758 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ea052aec744b48778ad23327985ed5a8'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:19.741396 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-f3fe7228-afa0-43d9-aabb-ba3c11a9f758 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ea052aec744b48778ad23327985ed5a8'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:19.742071 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-f3fe7228-afa0-43d9-aabb-ba3c11a9f758 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ea052aec744b48778ad23327985ed5a8'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:19.742786 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-f3fe7228-afa0-43d9-aabb-ba3c11a9f758 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ea052aec744b48778ad23327985ed5a8'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:19.762966 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1251/2509] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:41:19 2022] DELETE /identity/v3/users/ea052aec744b48778ad23327985ed5a8 => generated 0 bytes in 247 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:41:20.057284 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-662c6ea7-b9c4-46d8-8f0f-bc21885295a1 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:20.058809 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-662c6ea7-b9c4-46d8-8f0f-bc21885295a1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:20.075699 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-662c6ea7-b9c4-46d8-8f0f-bc21885295a1 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:20.136012 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b05e852f-c012-4f3e-8f4b-24d30042a403 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:20.136653 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b05e852f-c012-4f3e-8f4b-24d30042a403 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:20.136935 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b05e852f-c012-4f3e-8f4b-24d30042a403 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:20.137214 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b05e852f-c012-4f3e-8f4b-24d30042a403 admin admin] PATH_INFO: `/v3/projects/044c3d8848984ba6a3d829883a1de041` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:20.138385 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b05e852f-c012-4f3e-8f4b-24d30042a403 admin admin] RBAC: Authorizing `identity:delete_project(project_id=044c3d8848984ba6a3d829883a1de041)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:20.140304 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b05e852f-c012-4f3e-8f4b-24d30042a403 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:20.175325 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:41:20.193279 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-b05e852f-c012-4f3e-8f4b-24d30042a403 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '044c3d8848984ba6a3d829883a1de041', 'request_id': 'req-b05e852f-c012-4f3e-8f4b-24d30042a403'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:20.200517 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-b05e852f-c012-4f3e-8f4b-24d30042a403 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '044c3d8848984ba6a3d829883a1de041', 'request_id': 'req-b05e852f-c012-4f3e-8f4b-24d30042a403'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:20.206721 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-b05e852f-c012-4f3e-8f4b-24d30042a403 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '044c3d8848984ba6a3d829883a1de041', 'request_id': 'req-b05e852f-c012-4f3e-8f4b-24d30042a403'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:20.219366 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-b05e852f-c012-4f3e-8f4b-24d30042a403 admin admin] The token cache is being invalidate because project 044c3d8848984ba6a3d829883a1de041 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:41:20.219657 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-b05e852f-c012-4f3e-8f4b-24d30042a403 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:20.220464 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-b05e852f-c012-4f3e-8f4b-24d30042a403 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:20.222007 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1259/2510] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:41:20 2022] DELETE /identity/v3/projects/044c3d8848984ba6a3d829883a1de041 => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:41:20.226844 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b399c4d-2321-4db8-96d8-5f8bc491890f None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:20.227132 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b399c4d-2321-4db8-96d8-5f8bc491890f None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:20.227409 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b399c4d-2321-4db8-96d8-5f8bc491890f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:20.247501 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-6b399c4d-2321-4db8-96d8-5f8bc491890f None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:41:20.248766 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-6b399c4d-2321-4db8-96d8-5f8bc491890f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:20.310456 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1252/2511] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:41:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:41:20.319362 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f08dba-91e9-4996-b449-0d9598fce4d4 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:20.319656 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f08dba-91e9-4996-b449-0d9598fce4d4 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:20.319942 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f08dba-91e9-4996-b449-0d9598fce4d4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:20.339940 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-49f08dba-91e9-4996-b449-0d9598fce4d4 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:41:20.341160 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-49f08dba-91e9-4996-b449-0d9598fce4d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:20.349220 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1260/2512] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:41:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 21:41:20.353659 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b399c4d-2321-4db8-96d8-5f8bc491890f None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:20.364033 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b399c4d-2321-4db8-96d8-5f8bc491890f None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:20.368968 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9772e9a1-9a61-47e4-8211-ef79876926b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:20.369572 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9772e9a1-9a61-47e4-8211-ef79876926b6 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:20.369883 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9772e9a1-9a61-47e4-8211-ef79876926b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:20.370160 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9772e9a1-9a61-47e4-8211-ef79876926b6 admin admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:20.370810 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9772e9a1-9a61-47e4-8211-ef79876926b6 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:41:20.371113 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9772e9a1-9a61-47e4-8211-ef79876926b6 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:20.372735 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9772e9a1-9a61-47e4-8211-ef79876926b6 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:20.385467 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1253/2513] 10.222.0.90 () {58 vars in 1153 bytes} [Mon Aug 29 21:41:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:41:20.389623 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49f08dba-91e9-4996-b449-0d9598fce4d4 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:20.398947 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49f08dba-91e9-4996-b449-0d9598fce4d4 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:20.403595 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-863e7e1b-77f5-4b34-9c75-79bd840b5c18 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:20.404189 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-863e7e1b-77f5-4b34-9c75-79bd840b5c18 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:20.404469 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-863e7e1b-77f5-4b34-9c75-79bd840b5c18 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:20.404742 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-863e7e1b-77f5-4b34-9c75-79bd840b5c18 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:20.405385 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-863e7e1b-77f5-4b34-9c75-79bd840b5c18 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:20.407389 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-863e7e1b-77f5-4b34-9c75-79bd840b5c18 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:20.434902 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1261/2514] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:41:20 2022] POST /identity/v3/projects => generated 383 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 21:41:20.439464 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9772e9a1-9a61-47e4-8211-ef79876926b6 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:20.449900 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9772e9a1-9a61-47e4-8211-ef79876926b6 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:20.508980 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b802a9a7-0535-4bba-8bb0-91e0840347be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:20.509598 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b802a9a7-0535-4bba-8bb0-91e0840347be admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:20.509914 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b802a9a7-0535-4bba-8bb0-91e0840347be admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:20.510197 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b802a9a7-0535-4bba-8bb0-91e0840347be admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:20.510851 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b802a9a7-0535-4bba-8bb0-91e0840347be admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:20.512778 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b802a9a7-0535-4bba-8bb0-91e0840347be admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:20.534246 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1254/2515] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:41:20 2022] POST /identity/v3/users => generated 292 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 21:41:20.538616 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-863e7e1b-77f5-4b34-9c75-79bd840b5c18 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:20.549446 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-863e7e1b-77f5-4b34-9c75-79bd840b5c18 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:20.554011 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03094888-bb48-421d-beee-4494a9d65c46 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:20.554853 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03094888-bb48-421d-beee-4494a9d65c46 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:20.555133 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03094888-bb48-421d-beee-4494a9d65c46 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:20.555405 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03094888-bb48-421d-beee-4494a9d65c46 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:20.556042 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03094888-bb48-421d-beee-4494a9d65c46 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:20.557622 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03094888-bb48-421d-beee-4494a9d65c46 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:20.567749 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1262/2516] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:41:20 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:41:20.571854 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b802a9a7-0535-4bba-8bb0-91e0840347be admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:20.582786 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b802a9a7-0535-4bba-8bb0-91e0840347be admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:20.587491 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b58cd854-502b-4c9e-b484-cb3c7354f655 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:20.588099 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b58cd854-502b-4c9e-b484-cb3c7354f655 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:20.588385 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b58cd854-502b-4c9e-b484-cb3c7354f655 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:20.588655 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b58cd854-502b-4c9e-b484-cb3c7354f655 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:20.589300 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b58cd854-502b-4c9e-b484-cb3c7354f655 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:20.590958 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b58cd854-502b-4c9e-b484-cb3c7354f655 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:20.603230 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1255/2517] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:41:20 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:41:20.607571 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03094888-bb48-421d-beee-4494a9d65c46 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:20.619122 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03094888-bb48-421d-beee-4494a9d65c46 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:20.624216 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68dfba39-cef6-4fc4-8eea-175302704ff8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:20.624883 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68dfba39-cef6-4fc4-8eea-175302704ff8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:20.625164 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68dfba39-cef6-4fc4-8eea-175302704ff8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:20.625440 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68dfba39-cef6-4fc4-8eea-175302704ff8 admin admin] PATH_INFO: `/v3/projects/661cadaf089140148a3d3938de50141c/users/fe52a28f47944ee7bf7985704aefb161/roles/02e242d5520740d4b77a5f4008c7eb57` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:20.640625 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68dfba39-cef6-4fc4-8eea-175302704ff8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=661cadaf089140148a3d3938de50141c, user_id=fe52a28f47944ee7bf7985704aefb161, role_id=02e242d5520740d4b77a5f4008c7eb57)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:20.644225 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68dfba39-cef6-4fc4-8eea-175302704ff8 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:20.657707 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1263/2518] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:41:20 2022] PUT /identity/v3/projects/661cadaf089140148a3d3938de50141c/users/fe52a28f47944ee7bf7985704aefb161/roles/02e242d5520740d4b77a5f4008c7eb57 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:41:20.668914 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4f6a3cd-8962-45d0-859f-e5652e943972 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:20.669444 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4f6a3cd-8962-45d0-859f-e5652e943972 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:20.669444 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4f6a3cd-8962-45d0-859f-e5652e943972 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:20.698802 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-f4f6a3cd-8962-45d0-859f-e5652e943972 None None] MFA Rules not processed for user `fe52a28f47944ee7bf7985704aefb161`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:41:20.700167 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-f4f6a3cd-8962-45d0-859f-e5652e943972 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:20.706010 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=91,keystone:DELETE=5,keystone:INSERT=4 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:41:20.794631 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1256/2519] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:41:20 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:41:20.806024 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68dfba39-cef6-4fc4-8eea-175302704ff8 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:20.808040 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-68dfba39-cef6-4fc4-8eea-175302704ff8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:20.874822 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68dfba39-cef6-4fc4-8eea-175302704ff8 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:20.880531 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8d05e1a-0bc1-4549-b922-656ca4772370 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dc226f7ddcb49ecacb51ce57fdfc11b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:20.881165 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8d05e1a-0bc1-4549-b922-656ca4772370 service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:20.881451 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8d05e1a-0bc1-4549-b922-656ca4772370 service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:20.881729 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8d05e1a-0bc1-4549-b922-656ca4772370 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:20.890601 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8d05e1a-0bc1-4549-b922-656ca4772370 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:20.892809 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8d05e1a-0bc1-4549-b922-656ca4772370 service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:20.902205 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1264/2520] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:41:20 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:41:20.921217 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4f6a3cd-8962-45d0-859f-e5652e943972 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:20.995865 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4f6a3cd-8962-45d0-859f-e5652e943972 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:21.000776 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18920a39-358b-455e-a375-9ac7ce6840b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:21.002250 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18920a39-358b-455e-a375-9ac7ce6840b5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:21.002538 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18920a39-358b-455e-a375-9ac7ce6840b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:21.002824 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18920a39-358b-455e-a375-9ac7ce6840b5 admin admin] PATH_INFO: `/v3/users/fe52a28f47944ee7bf7985704aefb161` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:21.004019 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18920a39-358b-455e-a375-9ac7ce6840b5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=fe52a28f47944ee7bf7985704aefb161)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:21.006020 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18920a39-358b-455e-a375-9ac7ce6840b5 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:21.058621 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-18920a39-358b-455e-a375-9ac7ce6840b5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fe52a28f47944ee7bf7985704aefb161'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:21.067222 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-18920a39-358b-455e-a375-9ac7ce6840b5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fe52a28f47944ee7bf7985704aefb161'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:21.078235 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-18920a39-358b-455e-a375-9ac7ce6840b5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fe52a28f47944ee7bf7985704aefb161'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:21.078877 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-18920a39-358b-455e-a375-9ac7ce6840b5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fe52a28f47944ee7bf7985704aefb161'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:21.079546 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-18920a39-358b-455e-a375-9ac7ce6840b5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fe52a28f47944ee7bf7985704aefb161'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:21.106886 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1257/2521] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:41:20 2022] DELETE /identity/v3/users/fe52a28f47944ee7bf7985704aefb161 => generated 0 bytes in 186 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:41:21.112030 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95b61958-dff9-401b-9ff4-e5bcfde8f347 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:21.112507 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95b61958-dff9-401b-9ff4-e5bcfde8f347 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:21.112600 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95b61958-dff9-401b-9ff4-e5bcfde8f347 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:21.133568 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-95b61958-dff9-401b-9ff4-e5bcfde8f347 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:41:21.134819 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-95b61958-dff9-401b-9ff4-e5bcfde8f347 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:21.185134 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=92,keystone:DELETE=5,keystone:INSERT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:41:21.199096 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1265/2522] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:41:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:41:21.209235 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18920a39-358b-455e-a375-9ac7ce6840b5 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:21.210884 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-18920a39-358b-455e-a375-9ac7ce6840b5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:21.255672 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:41:21.288944 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18920a39-358b-455e-a375-9ac7ce6840b5 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:21.292756 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-316f2a53-723e-464e-9f05-59a4a9ee3b1c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:21.293439 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-316f2a53-723e-464e-9f05-59a4a9ee3b1c service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:21.293724 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-316f2a53-723e-464e-9f05-59a4a9ee3b1c service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:21.294026 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-316f2a53-723e-464e-9f05-59a4a9ee3b1c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:21.304672 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-316f2a53-723e-464e-9f05-59a4a9ee3b1c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:21.306353 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-316f2a53-723e-464e-9f05-59a4a9ee3b1c service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:21.317103 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1258/2523] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:41:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:41:21.618259 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95b61958-dff9-401b-9ff4-e5bcfde8f347 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:21.619799 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-95b61958-dff9-401b-9ff4-e5bcfde8f347 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:21.637550 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95b61958-dff9-401b-9ff4-e5bcfde8f347 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:21.644628 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32b59704-b39b-4968-a788-38f5f700b230 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:21.645252 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32b59704-b39b-4968-a788-38f5f700b230 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:21.645554 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32b59704-b39b-4968-a788-38f5f700b230 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:21.645856 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32b59704-b39b-4968-a788-38f5f700b230 admin admin] PATH_INFO: `/v3/projects/661cadaf089140148a3d3938de50141c` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:21.647402 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32b59704-b39b-4968-a788-38f5f700b230 admin admin] RBAC: Authorizing `identity:delete_project(project_id=661cadaf089140148a3d3938de50141c)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:21.650029 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32b59704-b39b-4968-a788-38f5f700b230 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:21.713654 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-32b59704-b39b-4968-a788-38f5f700b230 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '661cadaf089140148a3d3938de50141c', 'request_id': 'req-32b59704-b39b-4968-a788-38f5f700b230'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:21.722028 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-32b59704-b39b-4968-a788-38f5f700b230 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '661cadaf089140148a3d3938de50141c', 'request_id': 'req-32b59704-b39b-4968-a788-38f5f700b230'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:21.730382 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-32b59704-b39b-4968-a788-38f5f700b230 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '661cadaf089140148a3d3938de50141c', 'request_id': 'req-32b59704-b39b-4968-a788-38f5f700b230'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:21.745609 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-32b59704-b39b-4968-a788-38f5f700b230 admin admin] The token cache is being invalidate because project 661cadaf089140148a3d3938de50141c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:41:21.745955 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-32b59704-b39b-4968-a788-38f5f700b230 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:21.749385 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-32b59704-b39b-4968-a788-38f5f700b230 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:21.751913 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1266/2524] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:41:21 2022] DELETE /identity/v3/projects/661cadaf089140148a3d3938de50141c => generated 0 bytes in 134 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:41:21.756604 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-509ec8ee-1c1c-4e74-b506-062737308b95 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:21.757307 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-509ec8ee-1c1c-4e74-b506-062737308b95 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:21.757307 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-509ec8ee-1c1c-4e74-b506-062737308b95 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:21.781145 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-509ec8ee-1c1c-4e74-b506-062737308b95 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:41:21.783162 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-509ec8ee-1c1c-4e74-b506-062737308b95 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:21.868533 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1259/2525] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:41:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:41:21.878067 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c3e295-ce81-44bf-a2aa-80677a5aff15 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:21.878067 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c3e295-ce81-44bf-a2aa-80677a5aff15 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:21.878067 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c3e295-ce81-44bf-a2aa-80677a5aff15 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:21.905478 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-21c3e295-ce81-44bf-a2aa-80677a5aff15 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:41:21.906950 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-21c3e295-ce81-44bf-a2aa-80677a5aff15 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:21.916372 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1267/2526] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:41:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:41:21.921089 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-509ec8ee-1c1c-4e74-b506-062737308b95 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:21.931362 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-509ec8ee-1c1c-4e74-b506-062737308b95 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:21.936520 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe5de5c7-cc48-472c-8200-ea883c13097e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:21.937140 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe5de5c7-cc48-472c-8200-ea883c13097e admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:21.937424 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe5de5c7-cc48-472c-8200-ea883c13097e admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:21.937700 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe5de5c7-cc48-472c-8200-ea883c13097e admin admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:21.938439 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe5de5c7-cc48-472c-8200-ea883c13097e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:41:21.938772 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe5de5c7-cc48-472c-8200-ea883c13097e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:21.940399 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe5de5c7-cc48-472c-8200-ea883c13097e admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:21.952335 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1260/2527] 10.222.0.90 () {58 vars in 1153 bytes} [Mon Aug 29 21:41:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:41:21.957625 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21c3e295-ce81-44bf-a2aa-80677a5aff15 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:21.966968 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21c3e295-ce81-44bf-a2aa-80677a5aff15 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:21.972373 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5ac0df6-5c6f-4b6e-a1f5-c7e2b499e410 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:21.972971 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5ac0df6-5c6f-4b6e-a1f5-c7e2b499e410 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:21.973276 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5ac0df6-5c6f-4b6e-a1f5-c7e2b499e410 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:21.973546 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5ac0df6-5c6f-4b6e-a1f5-c7e2b499e410 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:21.974229 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5ac0df6-5c6f-4b6e-a1f5-c7e2b499e410 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:21.976096 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5ac0df6-5c6f-4b6e-a1f5-c7e2b499e410 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:22.003643 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1268/2528] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:41:21 2022] POST /identity/v3/projects => generated 373 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:41:22.008137 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe5de5c7-cc48-472c-8200-ea883c13097e admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:22.023562 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe5de5c7-cc48-472c-8200-ea883c13097e admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:22.085028 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31201bb4-ad6e-432f-b704-5cf3ded71b9e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:22.085572 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31201bb4-ad6e-432f-b704-5cf3ded71b9e admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:22.085919 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31201bb4-ad6e-432f-b704-5cf3ded71b9e admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:22.086196 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31201bb4-ad6e-432f-b704-5cf3ded71b9e admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:22.086845 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31201bb4-ad6e-432f-b704-5cf3ded71b9e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:22.088764 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31201bb4-ad6e-432f-b704-5cf3ded71b9e admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:22.112046 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1261/2529] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:41:22 2022] POST /identity/v3/users => generated 287 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 21:41:22.116342 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5ac0df6-5c6f-4b6e-a1f5-c7e2b499e410 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:22.127455 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5ac0df6-5c6f-4b6e-a1f5-c7e2b499e410 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:22.132265 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d5225d3-a202-4054-9a0c-f9bfbad323c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:22.132866 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d5225d3-a202-4054-9a0c-f9bfbad323c0 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:22.133143 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d5225d3-a202-4054-9a0c-f9bfbad323c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:22.133416 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d5225d3-a202-4054-9a0c-f9bfbad323c0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:22.134085 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d5225d3-a202-4054-9a0c-f9bfbad323c0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:22.135664 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d5225d3-a202-4054-9a0c-f9bfbad323c0 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:22.148520 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1269/2530] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:41:22 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:41:22.152996 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31201bb4-ad6e-432f-b704-5cf3ded71b9e admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:22.163863 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31201bb4-ad6e-432f-b704-5cf3ded71b9e admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:22.168675 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a94ee84b-9c75-4b00-a36f-8c8061dfc9bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:22.169289 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a94ee84b-9c75-4b00-a36f-8c8061dfc9bc admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:22.169569 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a94ee84b-9c75-4b00-a36f-8c8061dfc9bc admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:22.169880 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a94ee84b-9c75-4b00-a36f-8c8061dfc9bc admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:22.170524 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a94ee84b-9c75-4b00-a36f-8c8061dfc9bc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:22.172120 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a94ee84b-9c75-4b00-a36f-8c8061dfc9bc admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:22.183940 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1262/2531] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:41:22 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:41:22.188150 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d5225d3-a202-4054-9a0c-f9bfbad323c0 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:22.199050 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d5225d3-a202-4054-9a0c-f9bfbad323c0 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:22.204014 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8478b2bc-7370-48b3-93f8-f31168dd4655 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:22.205551 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8478b2bc-7370-48b3-93f8-f31168dd4655 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:22.205859 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8478b2bc-7370-48b3-93f8-f31168dd4655 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:22.206135 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8478b2bc-7370-48b3-93f8-f31168dd4655 admin admin] PATH_INFO: `/v3/projects/131b91a381924890adc10e36d7493f77/users/360d757ce377409b8733924a471710b7/roles/02e242d5520740d4b77a5f4008c7eb57` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:22.220431 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8478b2bc-7370-48b3-93f8-f31168dd4655 admin admin] RBAC: Authorizing `identity:create_grant(project_id=131b91a381924890adc10e36d7493f77, user_id=360d757ce377409b8733924a471710b7, role_id=02e242d5520740d4b77a5f4008c7eb57)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:22.223175 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8478b2bc-7370-48b3-93f8-f31168dd4655 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:22.236493 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1270/2532] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:41:22 2022] PUT /identity/v3/projects/131b91a381924890adc10e36d7493f77/users/360d757ce377409b8733924a471710b7/roles/02e242d5520740d4b77a5f4008c7eb57 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:41:22.246721 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bad6bb7-a0fe-4510-8755-cde6c5e705bc None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:22.247039 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bad6bb7-a0fe-4510-8755-cde6c5e705bc None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:22.247317 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bad6bb7-a0fe-4510-8755-cde6c5e705bc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:22.278114 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-9bad6bb7-a0fe-4510-8755-cde6c5e705bc None None] MFA Rules not processed for user `360d757ce377409b8733924a471710b7`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:41:22.279349 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-9bad6bb7-a0fe-4510-8755-cde6c5e705bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:22.299615 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:41:22.347115 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1263/2533] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:41:22 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:41:22.358180 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8478b2bc-7370-48b3-93f8-f31168dd4655 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:22.359935 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-8478b2bc-7370-48b3-93f8-f31168dd4655 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:22.418225 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8478b2bc-7370-48b3-93f8-f31168dd4655 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:22.421854 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1bb8731-9725-4e06-b55e-38ca730f3c70 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dc226f7ddcb49ecacb51ce57fdfc11b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:22.422475 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1bb8731-9725-4e06-b55e-38ca730f3c70 service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:22.423326 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1bb8731-9725-4e06-b55e-38ca730f3c70 service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:22.423796 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1bb8731-9725-4e06-b55e-38ca730f3c70 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:22.432585 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1bb8731-9725-4e06-b55e-38ca730f3c70 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:22.434022 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1bb8731-9725-4e06-b55e-38ca730f3c70 service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:22.443412 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1271/2534] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:41:22 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:41:22.464558 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bad6bb7-a0fe-4510-8755-cde6c5e705bc None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:22.527815 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bad6bb7-a0fe-4510-8755-cde6c5e705bc None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:22.532894 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffc42bc6-0191-42ab-8821-4b676617e7ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:22.533567 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffc42bc6-0191-42ab-8821-4b676617e7ce admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:22.533884 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffc42bc6-0191-42ab-8821-4b676617e7ce admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:22.534180 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffc42bc6-0191-42ab-8821-4b676617e7ce admin admin] PATH_INFO: `/v3/users/360d757ce377409b8733924a471710b7` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:22.535401 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffc42bc6-0191-42ab-8821-4b676617e7ce admin admin] RBAC: Authorizing `identity:delete_user(user_id=360d757ce377409b8733924a471710b7)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:22.537260 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffc42bc6-0191-42ab-8821-4b676617e7ce admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:22.586849 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-ffc42bc6-0191-42ab-8821-4b676617e7ce admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '360d757ce377409b8733924a471710b7'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:22.596050 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-ffc42bc6-0191-42ab-8821-4b676617e7ce admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '360d757ce377409b8733924a471710b7'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:22.606053 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-ffc42bc6-0191-42ab-8821-4b676617e7ce admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '360d757ce377409b8733924a471710b7'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:22.606761 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-ffc42bc6-0191-42ab-8821-4b676617e7ce admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '360d757ce377409b8733924a471710b7'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:22.607642 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-ffc42bc6-0191-42ab-8821-4b676617e7ce admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '360d757ce377409b8733924a471710b7'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:22.633494 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1264/2535] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:41:22 2022] DELETE /identity/v3/users/360d757ce377409b8733924a471710b7 => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:41:22.638104 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab3f4b9d-a326-4b47-afff-3765717d2684 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:22.638391 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab3f4b9d-a326-4b47-afff-3765717d2684 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:22.638669 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab3f4b9d-a326-4b47-afff-3765717d2684 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:22.658596 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-ab3f4b9d-a326-4b47-afff-3765717d2684 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:41:22.660095 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-ab3f4b9d-a326-4b47-afff-3765717d2684 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:22.673348 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:41:22.724164 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1272/2536] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:41:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:41:22.734000 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffc42bc6-0191-42ab-8821-4b676617e7ce admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:22.736068 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-ffc42bc6-0191-42ab-8821-4b676617e7ce admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:22.786004 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:41:22.801879 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffc42bc6-0191-42ab-8821-4b676617e7ce admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:22.807568 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3402507-68e6-484b-813d-14bb2d69fb3c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:22.808202 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3402507-68e6-484b-813d-14bb2d69fb3c service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:22.808490 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3402507-68e6-484b-813d-14bb2d69fb3c service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:22.808766 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3402507-68e6-484b-813d-14bb2d69fb3c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:22.823165 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3402507-68e6-484b-813d-14bb2d69fb3c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:22.824659 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3402507-68e6-484b-813d-14bb2d69fb3c service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:22.834843 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1265/2537] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:41:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:41:23.126058 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab3f4b9d-a326-4b47-afff-3765717d2684 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:23.127880 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-ab3f4b9d-a326-4b47-afff-3765717d2684 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:23.140529 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab3f4b9d-a326-4b47-afff-3765717d2684 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:23.146384 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2892d8b-e9b8-4ccf-b0a0-88e7baac3366 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:23.147040 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2892d8b-e9b8-4ccf-b0a0-88e7baac3366 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:23.147329 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2892d8b-e9b8-4ccf-b0a0-88e7baac3366 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:23.147613 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2892d8b-e9b8-4ccf-b0a0-88e7baac3366 admin admin] PATH_INFO: `/v3/projects/131b91a381924890adc10e36d7493f77` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:23.148862 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2892d8b-e9b8-4ccf-b0a0-88e7baac3366 admin admin] RBAC: Authorizing `identity:delete_project(project_id=131b91a381924890adc10e36d7493f77)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:23.150820 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2892d8b-e9b8-4ccf-b0a0-88e7baac3366 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:23.210779 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-c2892d8b-e9b8-4ccf-b0a0-88e7baac3366 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '131b91a381924890adc10e36d7493f77', 'request_id': 'req-c2892d8b-e9b8-4ccf-b0a0-88e7baac3366'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:23.218403 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-c2892d8b-e9b8-4ccf-b0a0-88e7baac3366 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '131b91a381924890adc10e36d7493f77', 'request_id': 'req-c2892d8b-e9b8-4ccf-b0a0-88e7baac3366'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:23.229321 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-c2892d8b-e9b8-4ccf-b0a0-88e7baac3366 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '131b91a381924890adc10e36d7493f77', 'request_id': 'req-c2892d8b-e9b8-4ccf-b0a0-88e7baac3366'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:23.244522 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-c2892d8b-e9b8-4ccf-b0a0-88e7baac3366 admin admin] The token cache is being invalidate because project 131b91a381924890adc10e36d7493f77 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:41:23.244864 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-c2892d8b-e9b8-4ccf-b0a0-88e7baac3366 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:23.245560 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-c2892d8b-e9b8-4ccf-b0a0-88e7baac3366 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:23.247251 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1273/2538] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:41:23 2022] DELETE /identity/v3/projects/131b91a381924890adc10e36d7493f77 => generated 0 bytes in 122 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:41:23.252967 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0f85692-ec51-4b1d-9002-5e546d24618d None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:23.253257 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0f85692-ec51-4b1d-9002-5e546d24618d None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:23.253532 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0f85692-ec51-4b1d-9002-5e546d24618d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:23.274197 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-c0f85692-ec51-4b1d-9002-5e546d24618d None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:41:23.275644 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-c0f85692-ec51-4b1d-9002-5e546d24618d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:23.339350 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1266/2539] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:41:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 21:41:23.349125 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0cebb71-d47c-417e-8c1c-b7d0b90b93b5 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:23.349358 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0cebb71-d47c-417e-8c1c-b7d0b90b93b5 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:23.349624 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0cebb71-d47c-417e-8c1c-b7d0b90b93b5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:23.370156 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-d0cebb71-d47c-417e-8c1c-b7d0b90b93b5 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:41:23.371310 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-d0cebb71-d47c-417e-8c1c-b7d0b90b93b5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:23.380124 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1274/2540] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:41:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:41:23.442154 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0f85692-ec51-4b1d-9002-5e546d24618d None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:23.451828 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0f85692-ec51-4b1d-9002-5e546d24618d None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:23.456989 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d587f37b-f94c-4642-8c1a-2e36a7a687f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:23.458136 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d587f37b-f94c-4642-8c1a-2e36a7a687f0 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:23.458136 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d587f37b-f94c-4642-8c1a-2e36a7a687f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:23.458342 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d587f37b-f94c-4642-8c1a-2e36a7a687f0 admin admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:23.458956 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d587f37b-f94c-4642-8c1a-2e36a7a687f0 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:41:23.459272 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d587f37b-f94c-4642-8c1a-2e36a7a687f0 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:23.460849 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d587f37b-f94c-4642-8c1a-2e36a7a687f0 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:23.472471 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1267/2541] 10.222.0.90 () {58 vars in 1153 bytes} [Mon Aug 29 21:41:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:41:23.478033 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0cebb71-d47c-417e-8c1c-b7d0b90b93b5 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:23.488344 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0cebb71-d47c-417e-8c1c-b7d0b90b93b5 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:23.493338 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c94bc573-c7c5-4b98-bef9-93455f21ab40 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:23.493927 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c94bc573-c7c5-4b98-bef9-93455f21ab40 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:23.494308 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c94bc573-c7c5-4b98-bef9-93455f21ab40 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:23.494583 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c94bc573-c7c5-4b98-bef9-93455f21ab40 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:23.495229 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c94bc573-c7c5-4b98-bef9-93455f21ab40 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:23.497119 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c94bc573-c7c5-4b98-bef9-93455f21ab40 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:23.528564 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1275/2542] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:41:23 2022] POST /identity/v3/projects => generated 373 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 21:41:23.533410 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d587f37b-f94c-4642-8c1a-2e36a7a687f0 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:23.543453 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d587f37b-f94c-4642-8c1a-2e36a7a687f0 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:23.609389 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-126ce74f-e156-455b-821b-dcc752060d68 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:23.610032 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-126ce74f-e156-455b-821b-dcc752060d68 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:23.611109 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-126ce74f-e156-455b-821b-dcc752060d68 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:23.611369 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-126ce74f-e156-455b-821b-dcc752060d68 admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:23.612216 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-126ce74f-e156-455b-821b-dcc752060d68 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:23.614182 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-126ce74f-e156-455b-821b-dcc752060d68 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:23.647400 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1268/2543] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:41:23 2022] POST /identity/v3/users => generated 287 bytes in 112 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:41:23.648580 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c94bc573-c7c5-4b98-bef9-93455f21ab40 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:23.660106 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c94bc573-c7c5-4b98-bef9-93455f21ab40 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:23.664960 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e43de9cf-e3db-401b-ab69-0e60d0c1c9e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:23.666028 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e43de9cf-e3db-401b-ab69-0e60d0c1c9e5 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:23.666311 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e43de9cf-e3db-401b-ab69-0e60d0c1c9e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:23.666587 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e43de9cf-e3db-401b-ab69-0e60d0c1c9e5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:23.667232 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e43de9cf-e3db-401b-ab69-0e60d0c1c9e5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:23.668813 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e43de9cf-e3db-401b-ab69-0e60d0c1c9e5 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:23.683490 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1276/2544] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:41:23 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:41:23.684616 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-126ce74f-e156-455b-821b-dcc752060d68 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:23.695291 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-126ce74f-e156-455b-821b-dcc752060d68 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:23.705241 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-939d7e3e-45fa-4f0d-ba73-d8225e7c8648 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:23.705898 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-939d7e3e-45fa-4f0d-ba73-d8225e7c8648 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:23.706298 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-939d7e3e-45fa-4f0d-ba73-d8225e7c8648 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:23.706575 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-939d7e3e-45fa-4f0d-ba73-d8225e7c8648 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:23.707218 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-939d7e3e-45fa-4f0d-ba73-d8225e7c8648 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:23.708797 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-939d7e3e-45fa-4f0d-ba73-d8225e7c8648 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:23.721496 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1269/2545] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:41:23 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:41:23.725976 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e43de9cf-e3db-401b-ab69-0e60d0c1c9e5 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:23.736763 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e43de9cf-e3db-401b-ab69-0e60d0c1c9e5 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:23.745765 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ed7a940-c1ed-4da3-b33c-c6086b279c54 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:23.746570 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ed7a940-c1ed-4da3-b33c-c6086b279c54 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:23.746918 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ed7a940-c1ed-4da3-b33c-c6086b279c54 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:23.747262 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ed7a940-c1ed-4da3-b33c-c6086b279c54 admin admin] PATH_INFO: `/v3/projects/902c413d242b4ff7a556c7e487ef5e90/users/7952b085a8b94dcbbeb6a2817ad602e9/roles/02e242d5520740d4b77a5f4008c7eb57` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:23.765033 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ed7a940-c1ed-4da3-b33c-c6086b279c54 admin admin] RBAC: Authorizing `identity:create_grant(project_id=902c413d242b4ff7a556c7e487ef5e90, user_id=7952b085a8b94dcbbeb6a2817ad602e9, role_id=02e242d5520740d4b77a5f4008c7eb57)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:23.768007 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ed7a940-c1ed-4da3-b33c-c6086b279c54 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:23.786397 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1277/2546] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:41:23 2022] PUT /identity/v3/projects/902c413d242b4ff7a556c7e487ef5e90/users/7952b085a8b94dcbbeb6a2817ad602e9/roles/02e242d5520740d4b77a5f4008c7eb57 => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:41:23.791585 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec7f52c5-e955-4ce8-9b31-5ca6ab0b4bba None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:23.792107 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec7f52c5-e955-4ce8-9b31-5ca6ab0b4bba None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:23.792216 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec7f52c5-e955-4ce8-9b31-5ca6ab0b4bba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:23.813110 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-ec7f52c5-e955-4ce8-9b31-5ca6ab0b4bba None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:41:23.814764 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-ec7f52c5-e955-4ce8-9b31-5ca6ab0b4bba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:23.871841 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:41:23.887093 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1270/2547] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:41:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:41:23.897197 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ed7a940-c1ed-4da3-b33c-c6086b279c54 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:23.902956 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-0ed7a940-c1ed-4da3-b33c-c6086b279c54 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:23.947567 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:41:23.966379 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ed7a940-c1ed-4da3-b33c-c6086b279c54 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:23.972131 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a18338ad-d464-4675-8254-bf454b07c7fa service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:23.972738 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a18338ad-d464-4675-8254-bf454b07c7fa service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:23.973038 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a18338ad-d464-4675-8254-bf454b07c7fa service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:23.973361 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a18338ad-d464-4675-8254-bf454b07c7fa service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:23.983365 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a18338ad-d464-4675-8254-bf454b07c7fa service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:23.985304 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a18338ad-d464-4675-8254-bf454b07c7fa service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:23.996180 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1278/2548] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:41:23 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:41:24.680577 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08b566e7-1ee2-44f3-a7e2-caa2793f4a7b None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:24.680864 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08b566e7-1ee2-44f3-a7e2-caa2793f4a7b None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:24.681358 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08b566e7-1ee2-44f3-a7e2-caa2793f4a7b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:24.711158 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-08b566e7-1ee2-44f3-a7e2-caa2793f4a7b None None] MFA Rules not processed for user `7952b085a8b94dcbbeb6a2817ad602e9`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:41:24.712582 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-08b566e7-1ee2-44f3-a7e2-caa2793f4a7b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:24.783403 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1271/2549] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:41:24 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:41:24.794551 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a18338ad-d464-4675-8254-bf454b07c7fa service neutron] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:24.796054 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-a18338ad-d464-4675-8254-bf454b07c7fa service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:24.864870 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a18338ad-d464-4675-8254-bf454b07c7fa service neutron] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:24.868561 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da7c9eda-d7dd-4d1b-b023-cc1d9ffb2bc4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dc226f7ddcb49ecacb51ce57fdfc11b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:24.869178 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da7c9eda-d7dd-4d1b-b023-cc1d9ffb2bc4 service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:24.869461 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da7c9eda-d7dd-4d1b-b023-cc1d9ffb2bc4 service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:24.869741 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da7c9eda-d7dd-4d1b-b023-cc1d9ffb2bc4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:24.881166 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da7c9eda-d7dd-4d1b-b023-cc1d9ffb2bc4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:24.882764 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da7c9eda-d7dd-4d1b-b023-cc1d9ffb2bc4 service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:24.892961 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1279/2550] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:41:24 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:41:24.895824 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08b566e7-1ee2-44f3-a7e2-caa2793f4a7b None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:24.897333 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-08b566e7-1ee2-44f3-a7e2-caa2793f4a7b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:24.911543 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08b566e7-1ee2-44f3-a7e2-caa2793f4a7b None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:24.916591 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3f853c5-45b7-45a4-8542-1c5180a8870c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:24.917228 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3f853c5-45b7-45a4-8542-1c5180a8870c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:24.917514 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3f853c5-45b7-45a4-8542-1c5180a8870c admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:24.917890 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3f853c5-45b7-45a4-8542-1c5180a8870c admin admin] PATH_INFO: `/v3/users/af5249fcd973480ab998de75c4b19647` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:24.919086 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3f853c5-45b7-45a4-8542-1c5180a8870c admin admin] RBAC: Authorizing `identity:delete_user(user_id=af5249fcd973480ab998de75c4b19647)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:24.921007 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3f853c5-45b7-45a4-8542-1c5180a8870c admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:24.970291 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-b3f853c5-45b7-45a4-8542-1c5180a8870c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'af5249fcd973480ab998de75c4b19647'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:24.977911 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-b3f853c5-45b7-45a4-8542-1c5180a8870c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'af5249fcd973480ab998de75c4b19647'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:24.987286 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-b3f853c5-45b7-45a4-8542-1c5180a8870c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'af5249fcd973480ab998de75c4b19647'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:24.988006 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-b3f853c5-45b7-45a4-8542-1c5180a8870c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'af5249fcd973480ab998de75c4b19647'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:24.988653 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-b3f853c5-45b7-45a4-8542-1c5180a8870c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'af5249fcd973480ab998de75c4b19647'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:25.007563 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1272/2551] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:41:24 2022] DELETE /identity/v3/users/af5249fcd973480ab998de75c4b19647 => generated 0 bytes in 112 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:41:25.012064 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2182486a-8abf-4932-92f8-1ebe00d66dfc None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:25.012522 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2182486a-8abf-4932-92f8-1ebe00d66dfc None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:25.012804 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2182486a-8abf-4932-92f8-1ebe00d66dfc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:25.034270 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-2182486a-8abf-4932-92f8-1ebe00d66dfc None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:41:25.035604 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-2182486a-8abf-4932-92f8-1ebe00d66dfc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:25.102135 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1280/2552] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:41:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:41:25.112688 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3f853c5-45b7-45a4-8542-1c5180a8870c admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:25.114228 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-b3f853c5-45b7-45a4-8542-1c5180a8870c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:25.193010 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3f853c5-45b7-45a4-8542-1c5180a8870c admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:25.205416 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b929e69-3d41-4134-94ed-30f66dda000a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:25.205416 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b929e69-3d41-4134-94ed-30f66dda000a service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:25.205416 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b929e69-3d41-4134-94ed-30f66dda000a service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:25.205416 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b929e69-3d41-4134-94ed-30f66dda000a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:25.208175 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:41:25.211066 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b929e69-3d41-4134-94ed-30f66dda000a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:25.213145 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b929e69-3d41-4134-94ed-30f66dda000a service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:25.225267 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1273/2553] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:41:25 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 113 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 21:41:25.508923 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2182486a-8abf-4932-92f8-1ebe00d66dfc None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:25.510408 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-2182486a-8abf-4932-92f8-1ebe00d66dfc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:25.531065 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2182486a-8abf-4932-92f8-1ebe00d66dfc None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:25.536869 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9be0f0a-34c4-4c20-a660-da03104f7e38 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:25.537514 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9be0f0a-34c4-4c20-a660-da03104f7e38 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:25.537810 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9be0f0a-34c4-4c20-a660-da03104f7e38 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:25.543207 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9be0f0a-34c4-4c20-a660-da03104f7e38 admin admin] PATH_INFO: `/v3/projects/897f2055d8e14a58b2c85a47e64a6601` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:25.549773 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9be0f0a-34c4-4c20-a660-da03104f7e38 admin admin] RBAC: Authorizing `identity:delete_project(project_id=897f2055d8e14a58b2c85a47e64a6601)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:25.551751 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9be0f0a-34c4-4c20-a660-da03104f7e38 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:25.578684 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b929e69-3d41-4134-94ed-30f66dda000a service neutron] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:25.589717 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b929e69-3d41-4134-94ed-30f66dda000a service neutron] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:25.594102 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6f06d20-afb4-46d1-b1ca-92b6c9428fcd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:25.595674 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f06d20-afb4-46d1-b1ca-92b6c9428fcd service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:25.596487 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f06d20-afb4-46d1-b1ca-92b6c9428fcd service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:25.596790 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f06d20-afb4-46d1-b1ca-92b6c9428fcd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:25.598876 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-a6f06d20-afb4-46d1-b1ca-92b6c9428fcd service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:25.609252 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-c9be0f0a-34c4-4c20-a660-da03104f7e38 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '897f2055d8e14a58b2c85a47e64a6601', 'request_id': 'req-c9be0f0a-34c4-4c20-a660-da03104f7e38'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:25.612572 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6f06d20-afb4-46d1-b1ca-92b6c9428fcd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:25.614310 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6f06d20-afb4-46d1-b1ca-92b6c9428fcd service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:25.615754 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-c9be0f0a-34c4-4c20-a660-da03104f7e38 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '897f2055d8e14a58b2c85a47e64a6601', 'request_id': 'req-c9be0f0a-34c4-4c20-a660-da03104f7e38'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:25.621661 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-c9be0f0a-34c4-4c20-a660-da03104f7e38 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '897f2055d8e14a58b2c85a47e64a6601', 'request_id': 'req-c9be0f0a-34c4-4c20-a660-da03104f7e38'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:25.625499 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1274/2554] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:41:25 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 47 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 21:41:25.635395 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-c9be0f0a-34c4-4c20-a660-da03104f7e38 admin admin] The token cache is being invalidate because project 897f2055d8e14a58b2c85a47e64a6601 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:41:25.635763 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-c9be0f0a-34c4-4c20-a660-da03104f7e38 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:25.636447 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-c9be0f0a-34c4-4c20-a660-da03104f7e38 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:25.638052 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1281/2555] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:41:25 2022] DELETE /identity/v3/projects/897f2055d8e14a58b2c85a47e64a6601 => generated 0 bytes in 130 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:41:25.643926 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6065dd4b-4f86-4ddf-9fb3-9d95f92dbef6 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:25.644351 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6065dd4b-4f86-4ddf-9fb3-9d95f92dbef6 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:25.644638 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6065dd4b-4f86-4ddf-9fb3-9d95f92dbef6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:25.665137 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-6065dd4b-4f86-4ddf-9fb3-9d95f92dbef6 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:41:25.666405 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-6065dd4b-4f86-4ddf-9fb3-9d95f92dbef6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:25.752066 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1275/2556] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:41:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:41:25.760093 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efbce4ea-1455-416d-8444-89cc6a33726b None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:25.760376 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efbce4ea-1455-416d-8444-89cc6a33726b None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:25.760649 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efbce4ea-1455-416d-8444-89cc6a33726b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:25.776091 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:41:25.783618 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-efbce4ea-1455-416d-8444-89cc6a33726b None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:41:25.786338 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-efbce4ea-1455-416d-8444-89cc6a33726b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:25.795596 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1282/2557] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:41:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:41:25.799976 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6065dd4b-4f86-4ddf-9fb3-9d95f92dbef6 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:25.810614 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6065dd4b-4f86-4ddf-9fb3-9d95f92dbef6 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:25.816029 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b3f986b-6f9d-4da9-96ea-251459f69fe1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:25.816648 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b3f986b-6f9d-4da9-96ea-251459f69fe1 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:25.816932 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b3f986b-6f9d-4da9-96ea-251459f69fe1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:25.817270 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b3f986b-6f9d-4da9-96ea-251459f69fe1 admin admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:25.817958 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b3f986b-6f9d-4da9-96ea-251459f69fe1 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:41:25.818294 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b3f986b-6f9d-4da9-96ea-251459f69fe1 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:25.819909 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b3f986b-6f9d-4da9-96ea-251459f69fe1 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:25.834660 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1276/2558] 10.222.0.90 () {58 vars in 1153 bytes} [Mon Aug 29 21:41:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:41:25.839843 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efbce4ea-1455-416d-8444-89cc6a33726b None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:25.849745 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efbce4ea-1455-416d-8444-89cc6a33726b None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:25.858483 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0061245d-f2b4-42c4-b685-7cc30bcd0073 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:25.859230 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0061245d-f2b4-42c4-b685-7cc30bcd0073 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:25.859590 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0061245d-f2b4-42c4-b685-7cc30bcd0073 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:25.859938 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0061245d-f2b4-42c4-b685-7cc30bcd0073 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:25.860808 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0061245d-f2b4-42c4-b685-7cc30bcd0073 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:25.862848 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0061245d-f2b4-42c4-b685-7cc30bcd0073 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:25.893535 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1283/2559] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:41:25 2022] POST /identity/v3/projects => generated 373 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:41:25.897779 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b3f986b-6f9d-4da9-96ea-251459f69fe1 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:25.908278 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b3f986b-6f9d-4da9-96ea-251459f69fe1 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:25.968569 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37769fb5-172f-414d-a025-528bf75d779d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:25.969184 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37769fb5-172f-414d-a025-528bf75d779d admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:25.969463 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37769fb5-172f-414d-a025-528bf75d779d admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:25.969733 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37769fb5-172f-414d-a025-528bf75d779d admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:25.970585 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37769fb5-172f-414d-a025-528bf75d779d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:25.972546 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37769fb5-172f-414d-a025-528bf75d779d admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:25.997862 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1277/2560] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:41:25 2022] POST /identity/v3/users => generated 287 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:41:25.999333 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0061245d-f2b4-42c4-b685-7cc30bcd0073 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:26.010335 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0061245d-f2b4-42c4-b685-7cc30bcd0073 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:26.015327 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67771428-aa05-4ee4-bc24-96b3f57d502a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:26.015965 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67771428-aa05-4ee4-bc24-96b3f57d502a admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:26.016375 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67771428-aa05-4ee4-bc24-96b3f57d502a admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:26.016661 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67771428-aa05-4ee4-bc24-96b3f57d502a admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:26.017899 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67771428-aa05-4ee4-bc24-96b3f57d502a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:26.019574 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67771428-aa05-4ee4-bc24-96b3f57d502a admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:26.030312 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1284/2561] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:41:25 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 29 21:41:26.034426 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37769fb5-172f-414d-a025-528bf75d779d admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:26.044652 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37769fb5-172f-414d-a025-528bf75d779d admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:26.049560 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66d30c2c-4bc7-421d-bfbd-3053db784496 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:26.050255 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d30c2c-4bc7-421d-bfbd-3053db784496 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:26.050539 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d30c2c-4bc7-421d-bfbd-3053db784496 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:26.050816 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d30c2c-4bc7-421d-bfbd-3053db784496 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:26.051587 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66d30c2c-4bc7-421d-bfbd-3053db784496 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:26.053203 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66d30c2c-4bc7-421d-bfbd-3053db784496 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:26.066202 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1278/2562] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:41:26 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:41:26.070513 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67771428-aa05-4ee4-bc24-96b3f57d502a admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:26.081752 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67771428-aa05-4ee4-bc24-96b3f57d502a admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:26.086622 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9579b16f-3bc2-4028-92ee-445d37d7b4c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:26.087292 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9579b16f-3bc2-4028-92ee-445d37d7b4c0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:26.087577 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9579b16f-3bc2-4028-92ee-445d37d7b4c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:26.087854 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9579b16f-3bc2-4028-92ee-445d37d7b4c0 admin admin] PATH_INFO: `/v3/projects/809fa9baaebb4ac79d20d50288e41804/users/89b4ac9ecb0c4358b8c0b1fbd44c89db/roles/02e242d5520740d4b77a5f4008c7eb57` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:26.103765 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9579b16f-3bc2-4028-92ee-445d37d7b4c0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=809fa9baaebb4ac79d20d50288e41804, user_id=89b4ac9ecb0c4358b8c0b1fbd44c89db, role_id=02e242d5520740d4b77a5f4008c7eb57)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:26.106484 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9579b16f-3bc2-4028-92ee-445d37d7b4c0 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:26.121252 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1285/2563] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:41:26 2022] PUT /identity/v3/projects/809fa9baaebb4ac79d20d50288e41804/users/89b4ac9ecb0c4358b8c0b1fbd44c89db/roles/02e242d5520740d4b77a5f4008c7eb57 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:41:26.132858 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74980ed6-4324-4066-b4e6-061045a59f46 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:26.133256 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74980ed6-4324-4066-b4e6-061045a59f46 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:26.133607 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74980ed6-4324-4066-b4e6-061045a59f46 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:26.162678 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-74980ed6-4324-4066-b4e6-061045a59f46 None None] MFA Rules not processed for user `89b4ac9ecb0c4358b8c0b1fbd44c89db`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:41:26.163901 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-74980ed6-4324-4066-b4e6-061045a59f46 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:26.205424 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:41:26.243734 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1279/2564] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:41:26 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:41:26.246435 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9579b16f-3bc2-4028-92ee-445d37d7b4c0 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:26.270422 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31fb3079-4916-4a55-846d-60c4b54e6bb0 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:26.270719 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31fb3079-4916-4a55-846d-60c4b54e6bb0 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:26.270997 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31fb3079-4916-4a55-846d-60c4b54e6bb0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:26.292484 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-31fb3079-4916-4a55-846d-60c4b54e6bb0 None None] MFA Rules not processed for user `7952b085a8b94dcbbeb6a2817ad602e9`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:41:26.294129 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-31fb3079-4916-4a55-846d-60c4b54e6bb0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:26.320202 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9579b16f-3bc2-4028-92ee-445d37d7b4c0 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:26.326482 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dd0bed2-75d7-48db-9c87-e8e202f2c976 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:26.327101 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dd0bed2-75d7-48db-9c87-e8e202f2c976 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:26.327385 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dd0bed2-75d7-48db-9c87-e8e202f2c976 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:26.327915 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dd0bed2-75d7-48db-9c87-e8e202f2c976 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:26.328550 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dd0bed2-75d7-48db-9c87-e8e202f2c976 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:26.332409 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dd0bed2-75d7-48db-9c87-e8e202f2c976 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:26.363926 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1280/2565] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:41:26 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 21:41:26.365212 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1286/2566] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:41:26 2022] POST /identity/v3/projects => generated 373 bytes in 119 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:41:26.368911 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31fb3079-4916-4a55-846d-60c4b54e6bb0 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:26.382105 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31fb3079-4916-4a55-846d-60c4b54e6bb0 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:26.451291 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f2f819b-5a5d-49da-9c63-0983d69329cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:26.451986 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f2f819b-5a5d-49da-9c63-0983d69329cf admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:26.452447 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f2f819b-5a5d-49da-9c63-0983d69329cf admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:26.453091 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f2f819b-5a5d-49da-9c63-0983d69329cf admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:26.453880 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f2f819b-5a5d-49da-9c63-0983d69329cf admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:26.459575 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f2f819b-5a5d-49da-9c63-0983d69329cf admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:26.484327 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1281/2567] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:41:26 2022] POST /identity/v3/users => generated 293 bytes in 116 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:41:26.488890 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dd0bed2-75d7-48db-9c87-e8e202f2c976 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:26.504720 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dd0bed2-75d7-48db-9c87-e8e202f2c976 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:26.510957 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66250935-ffbb-44e1-9ca0-d55ec3fc35a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:26.511950 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66250935-ffbb-44e1-9ca0-d55ec3fc35a2 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:26.511950 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66250935-ffbb-44e1-9ca0-d55ec3fc35a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:26.512146 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66250935-ffbb-44e1-9ca0-d55ec3fc35a2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:26.512761 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66250935-ffbb-44e1-9ca0-d55ec3fc35a2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:26.514531 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66250935-ffbb-44e1-9ca0-d55ec3fc35a2 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:26.530475 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1287/2568] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:41:26 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:41:26.535976 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f2f819b-5a5d-49da-9c63-0983d69329cf admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:26.546446 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f2f819b-5a5d-49da-9c63-0983d69329cf admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:26.552567 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c991cd44-b3b1-475b-9af3-be71fa56377f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:26.553258 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c991cd44-b3b1-475b-9af3-be71fa56377f admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:26.553539 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c991cd44-b3b1-475b-9af3-be71fa56377f admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:26.553843 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c991cd44-b3b1-475b-9af3-be71fa56377f admin admin] PATH_INFO: `/v3/projects/d541a155be98406ead40fa199d8cca56/users/4f5c4879949a49d584c4fa3c0cce6925/roles/a6e562cdc96c4be691d638990f8403fd` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:26.577401 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c991cd44-b3b1-475b-9af3-be71fa56377f admin admin] RBAC: Authorizing `identity:create_grant(project_id=d541a155be98406ead40fa199d8cca56, user_id=4f5c4879949a49d584c4fa3c0cce6925, role_id=a6e562cdc96c4be691d638990f8403fd)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:26.581035 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c991cd44-b3b1-475b-9af3-be71fa56377f admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:26.598947 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1282/2569] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:41:26 2022] PUT /identity/v3/projects/d541a155be98406ead40fa199d8cca56/users/4f5c4879949a49d584c4fa3c0cce6925/roles/a6e562cdc96c4be691d638990f8403fd => generated 0 bytes in 63 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 21:41:26.602815 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66250935-ffbb-44e1-9ca0-d55ec3fc35a2 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:26.618825 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66250935-ffbb-44e1-9ca0-d55ec3fc35a2 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:26.709310 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a58faca-f6d6-43be-b62f-86888a1b9e13 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:26.710257 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a58faca-f6d6-43be-b62f-86888a1b9e13 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:26.710724 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a58faca-f6d6-43be-b62f-86888a1b9e13 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:26.711156 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a58faca-f6d6-43be-b62f-86888a1b9e13 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:26.711969 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a58faca-f6d6-43be-b62f-86888a1b9e13 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:26.714073 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a58faca-f6d6-43be-b62f-86888a1b9e13 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:26.726357 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1288/2570] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:41:26 2022] GET /identity/v3/roles => generated 1420 bytes in 124 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:41:26.730706 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c991cd44-b3b1-475b-9af3-be71fa56377f admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:26.749631 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c991cd44-b3b1-475b-9af3-be71fa56377f admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:26.755971 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10c9fa19-04b8-4b68-8bdf-e672e7349d7e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:26.757361 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10c9fa19-04b8-4b68-8bdf-e672e7349d7e admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:26.757754 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10c9fa19-04b8-4b68-8bdf-e672e7349d7e admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:26.758160 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10c9fa19-04b8-4b68-8bdf-e672e7349d7e admin admin] PATH_INFO: `/v3/projects/d541a155be98406ead40fa199d8cca56/users/4f5c4879949a49d584c4fa3c0cce6925/roles/a6e562cdc96c4be691d638990f8403fd` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:26.761080 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10c9fa19-04b8-4b68-8bdf-e672e7349d7e admin admin] RBAC: Authorizing `identity:create_grant(project_id=d541a155be98406ead40fa199d8cca56, user_id=4f5c4879949a49d584c4fa3c0cce6925, role_id=a6e562cdc96c4be691d638990f8403fd)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:26.764991 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10c9fa19-04b8-4b68-8bdf-e672e7349d7e admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:26.791090 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1283/2571] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:41:26 2022] PUT /identity/v3/projects/d541a155be98406ead40fa199d8cca56/users/4f5c4879949a49d584c4fa3c0cce6925/roles/a6e562cdc96c4be691d638990f8403fd => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:41:26.792779 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14027dab-557a-4a35-94e2-242b4d3e1fab None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:26.793377 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14027dab-557a-4a35-94e2-242b4d3e1fab None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:26.797353 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14027dab-557a-4a35-94e2-242b4d3e1fab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:26.826506 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:41:26.832435 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-14027dab-557a-4a35-94e2-242b4d3e1fab None None] MFA Rules not processed for user `4f5c4879949a49d584c4fa3c0cce6925`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:41:26.836527 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-14027dab-557a-4a35-94e2-242b4d3e1fab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:26.937322 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1289/2572] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:41:26 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 146 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:41:26.949455 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10c9fa19-04b8-4b68-8bdf-e672e7349d7e admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:26.951658 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-10c9fa19-04b8-4b68-8bdf-e672e7349d7e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:27.005953 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:41:27.035645 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10c9fa19-04b8-4b68-8bdf-e672e7349d7e admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:27.040632 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6edf8a31-0169-4c81-8a08-e1018e152ca8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dc226f7ddcb49ecacb51ce57fdfc11b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:27.041451 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6edf8a31-0169-4c81-8a08-e1018e152ca8 service cinder] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:27.041937 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6edf8a31-0169-4c81-8a08-e1018e152ca8 service cinder] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:27.043065 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6edf8a31-0169-4c81-8a08-e1018e152ca8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:27.114712 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6edf8a31-0169-4c81-8a08-e1018e152ca8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:27.116170 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6edf8a31-0169-4c81-8a08-e1018e152ca8 service cinder] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:27.127062 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1284/2573] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:41:26 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 178 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:41:29.318541 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14027dab-557a-4a35-94e2-242b4d3e1fab None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:29.320836 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-14027dab-557a-4a35-94e2-242b4d3e1fab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:29.420149 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6edf8a31-0169-4c81-8a08-e1018e152ca8 service cinder] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:29.424238 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-6edf8a31-0169-4c81-8a08-e1018e152ca8 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:29.512461 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14027dab-557a-4a35-94e2-242b4d3e1fab None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:29.520904 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0669e44-e5ce-4753-b84c-36e93b3c7a26 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b95fa07a7c0549dbaac49406e2639e21', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:29.522295 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0669e44-e5ce-4753-b84c-36e93b3c7a26 service placement] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:29.522685 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0669e44-e5ce-4753-b84c-36e93b3c7a26 service placement] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:29.523086 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0669e44-e5ce-4753-b84c-36e93b3c7a26 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:29.525377 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-b0669e44-e5ce-4753-b84c-36e93b3c7a26 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:29.542592 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0669e44-e5ce-4753-b84c-36e93b3c7a26 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:29.544298 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0669e44-e5ce-4753-b84c-36e93b3c7a26 service placement] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:29.557778 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1290/2574] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:41:29 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 240 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:41:29.596449 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6edf8a31-0169-4c81-8a08-e1018e152ca8 service cinder] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:29.604386 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-487fa0e7-d1c0-4504-b723-880db7541933 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:29.604978 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-487fa0e7-d1c0-4504-b723-880db7541933 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:29.605419 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-487fa0e7-d1c0-4504-b723-880db7541933 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:29.605671 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-487fa0e7-d1c0-4504-b723-880db7541933 admin admin] PATH_INFO: `/v3/users/bbbfb39899b04668ac1b7ff28d18e839` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:29.607078 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-487fa0e7-d1c0-4504-b723-880db7541933 admin admin] RBAC: Authorizing `identity:delete_user(user_id=bbbfb39899b04668ac1b7ff28d18e839)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:29.609058 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-487fa0e7-d1c0-4504-b723-880db7541933 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:29.666930 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=93,keystone:DELETE=7 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:41:29.668874 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-487fa0e7-d1c0-4504-b723-880db7541933 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'bbbfb39899b04668ac1b7ff28d18e839'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:29.680246 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-487fa0e7-d1c0-4504-b723-880db7541933 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'bbbfb39899b04668ac1b7ff28d18e839'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:29.689218 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-487fa0e7-d1c0-4504-b723-880db7541933 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'bbbfb39899b04668ac1b7ff28d18e839'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:29.690125 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-487fa0e7-d1c0-4504-b723-880db7541933 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'bbbfb39899b04668ac1b7ff28d18e839'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:29.690828 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-487fa0e7-d1c0-4504-b723-880db7541933 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'bbbfb39899b04668ac1b7ff28d18e839'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:29.713020 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1285/2575] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:41:29 2022] DELETE /identity/v3/users/bbbfb39899b04668ac1b7ff28d18e839 => generated 0 bytes in 294 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:41:29.922128 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0669e44-e5ce-4753-b84c-36e93b3c7a26 service placement] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:29.946242 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0669e44-e5ce-4753-b84c-36e93b3c7a26 service placement] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:30.002504 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-487fa0e7-d1c0-4504-b723-880db7541933 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:30.004883 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-487fa0e7-d1c0-4504-b723-880db7541933 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:30.005742 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:41:30.028822 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9294d6bf-e9bf-4b03-b65e-79bdbb2ab006 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:30.029465 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9294d6bf-e9bf-4b03-b65e-79bdbb2ab006 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:30.029746 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9294d6bf-e9bf-4b03-b65e-79bdbb2ab006 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:30.030392 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9294d6bf-e9bf-4b03-b65e-79bdbb2ab006 admin admin] PATH_INFO: `/v3/projects/d3c829bce41c4be9a0027076ef0ace75` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:30.031232 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9294d6bf-e9bf-4b03-b65e-79bdbb2ab006 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d3c829bce41c4be9a0027076ef0ace75)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:30.033604 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9294d6bf-e9bf-4b03-b65e-79bdbb2ab006 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:30.100910 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-487fa0e7-d1c0-4504-b723-880db7541933 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:30.123396 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-9294d6bf-e9bf-4b03-b65e-79bdbb2ab006 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd3c829bce41c4be9a0027076ef0ace75', 'request_id': 'req-9294d6bf-e9bf-4b03-b65e-79bdbb2ab006'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:30.137401 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-9294d6bf-e9bf-4b03-b65e-79bdbb2ab006 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd3c829bce41c4be9a0027076ef0ace75', 'request_id': 'req-9294d6bf-e9bf-4b03-b65e-79bdbb2ab006'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:30.145664 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-9294d6bf-e9bf-4b03-b65e-79bdbb2ab006 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd3c829bce41c4be9a0027076ef0ace75', 'request_id': 'req-9294d6bf-e9bf-4b03-b65e-79bdbb2ab006'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:30.161869 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-9294d6bf-e9bf-4b03-b65e-79bdbb2ab006 admin admin] The token cache is being invalidate because project d3c829bce41c4be9a0027076ef0ace75 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:41:30.162224 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-9294d6bf-e9bf-4b03-b65e-79bdbb2ab006 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:30.162951 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-9294d6bf-e9bf-4b03-b65e-79bdbb2ab006 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:30.164842 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1291/2576] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:41:29 2022] DELETE /identity/v3/projects/d3c829bce41c4be9a0027076ef0ace75 => generated 0 bytes in 243 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:41:30.170377 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe7115fa-20b5-4a84-9824-f1d64b2885b5 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:30.170718 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe7115fa-20b5-4a84-9824-f1d64b2885b5 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:30.171842 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe7115fa-20b5-4a84-9824-f1d64b2885b5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:30.179269 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fd87b1a-acab-4a58-be6e-8284886d8832 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:30.180091 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fd87b1a-acab-4a58-be6e-8284886d8832 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:30.180229 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fd87b1a-acab-4a58-be6e-8284886d8832 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:30.180466 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fd87b1a-acab-4a58-be6e-8284886d8832 admin admin] PATH_INFO: `/v3/users/89b4ac9ecb0c4358b8c0b1fbd44c89db` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:30.181826 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fd87b1a-acab-4a58-be6e-8284886d8832 admin admin] RBAC: Authorizing `identity:delete_user(user_id=89b4ac9ecb0c4358b8c0b1fbd44c89db)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:30.184247 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fd87b1a-acab-4a58-be6e-8284886d8832 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:30.255824 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-2fd87b1a-acab-4a58-be6e-8284886d8832 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '89b4ac9ecb0c4358b8c0b1fbd44c89db'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:30.261308 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-fe7115fa-20b5-4a84-9824-f1d64b2885b5 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:41:30.264042 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-fe7115fa-20b5-4a84-9824-f1d64b2885b5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:30.264190 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-2fd87b1a-acab-4a58-be6e-8284886d8832 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '89b4ac9ecb0c4358b8c0b1fbd44c89db'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:30.274664 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-2fd87b1a-acab-4a58-be6e-8284886d8832 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '89b4ac9ecb0c4358b8c0b1fbd44c89db'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:30.275700 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-2fd87b1a-acab-4a58-be6e-8284886d8832 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '89b4ac9ecb0c4358b8c0b1fbd44c89db'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:30.276581 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-2fd87b1a-acab-4a58-be6e-8284886d8832 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '89b4ac9ecb0c4358b8c0b1fbd44c89db'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:30.284550 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1292/2577] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:41:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 21:41:30.291212 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=86,keystone:INSERT=2,keystone:DELETE=12 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:41:30.293038 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deec9574-0d72-4c20-a51e-9b14979122d3 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:30.293337 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deec9574-0d72-4c20-a51e-9b14979122d3 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:30.293698 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deec9574-0d72-4c20-a51e-9b14979122d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:30.309565 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1286/2578] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:41:30 2022] DELETE /identity/v3/users/89b4ac9ecb0c4358b8c0b1fbd44c89db => generated 0 bytes in 308 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 21:41:30.314567 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30492d6f-282e-461a-8bf6-7f4b439c5c19 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:30.314865 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30492d6f-282e-461a-8bf6-7f4b439c5c19 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:30.315144 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30492d6f-282e-461a-8bf6-7f4b439c5c19 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:30.318974 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-deec9574-0d72-4c20-a51e-9b14979122d3 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:41:30.320319 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-deec9574-0d72-4c20-a51e-9b14979122d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:30.338188 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-30492d6f-282e-461a-8bf6-7f4b439c5c19 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:41:30.339437 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-30492d6f-282e-461a-8bf6-7f4b439c5c19 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:30.393932 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1293/2579] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:41:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:41:30.399331 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deec9574-0d72-4c20-a51e-9b14979122d3 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:30.414551 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deec9574-0d72-4c20-a51e-9b14979122d3 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:30.422082 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1287/2580] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:41:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:41:30.422748 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a45a1565-9665-4f13-9d78-430a3bc61811 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:30.423375 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a45a1565-9665-4f13-9d78-430a3bc61811 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:30.423684 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a45a1565-9665-4f13-9d78-430a3bc61811 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:30.424141 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a45a1565-9665-4f13-9d78-430a3bc61811 admin admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:30.425324 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a45a1565-9665-4f13-9d78-430a3bc61811 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:41:30.425580 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a45a1565-9665-4f13-9d78-430a3bc61811 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:30.428051 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a45a1565-9665-4f13-9d78-430a3bc61811 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:30.434037 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30492d6f-282e-461a-8bf6-7f4b439c5c19 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:30.436289 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-30492d6f-282e-461a-8bf6-7f4b439c5c19 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:30.446065 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1294/2581] 10.222.0.90 () {58 vars in 1153 bytes} [Mon Aug 29 21:41:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:41:30.450232 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a45a1565-9665-4f13-9d78-430a3bc61811 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:30.464684 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a45a1565-9665-4f13-9d78-430a3bc61811 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:30.469596 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18f6da21-79f9-4258-9a8a-cb3fb32cb3e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:30.470564 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18f6da21-79f9-4258-9a8a-cb3fb32cb3e9 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:30.470844 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18f6da21-79f9-4258-9a8a-cb3fb32cb3e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:30.471118 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18f6da21-79f9-4258-9a8a-cb3fb32cb3e9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:30.471757 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18f6da21-79f9-4258-9a8a-cb3fb32cb3e9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:30.473961 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18f6da21-79f9-4258-9a8a-cb3fb32cb3e9 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:30.503659 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:41:30.516386 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1295/2582] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:41:30 2022] POST /identity/v3/projects => generated 383 bytes in 67 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 21:41:30.522623 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18f6da21-79f9-4258-9a8a-cb3fb32cb3e9 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:30.535330 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30492d6f-282e-461a-8bf6-7f4b439c5c19 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:30.618185 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c27755d-f78e-4a48-9ec8-c60af4f38be7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:30.622155 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c27755d-f78e-4a48-9ec8-c60af4f38be7 service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:30.622540 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c27755d-f78e-4a48-9ec8-c60af4f38be7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:30.622861 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c27755d-f78e-4a48-9ec8-c60af4f38be7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:30.637326 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c27755d-f78e-4a48-9ec8-c60af4f38be7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:30.639209 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c27755d-f78e-4a48-9ec8-c60af4f38be7 service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:30.640398 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18f6da21-79f9-4258-9a8a-cb3fb32cb3e9 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:30.648997 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c584062f-a2f4-46be-9a5c-da592c14efbd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:30.649747 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c584062f-a2f4-46be-9a5c-da592c14efbd admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:30.650163 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c584062f-a2f4-46be-9a5c-da592c14efbd admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:30.650530 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c584062f-a2f4-46be-9a5c-da592c14efbd admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:30.651234 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c584062f-a2f4-46be-9a5c-da592c14efbd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:30.653544 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c584062f-a2f4-46be-9a5c-da592c14efbd admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:30.654955 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1288/2583] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:41:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 221 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:41:30.687438 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1296/2584] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:41:30 2022] POST /identity/v3/users => generated 292 bytes in 165 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:41:30.689884 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c27755d-f78e-4a48-9ec8-c60af4f38be7 service neutron] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:30.707056 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c27755d-f78e-4a48-9ec8-c60af4f38be7 service neutron] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:30.708633 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da843e22-9aa7-4320-98e7-c0cc38d484a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:30.709388 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da843e22-9aa7-4320-98e7-c0cc38d484a8 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:30.709705 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da843e22-9aa7-4320-98e7-c0cc38d484a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:30.710202 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da843e22-9aa7-4320-98e7-c0cc38d484a8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:30.710862 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da843e22-9aa7-4320-98e7-c0cc38d484a8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:30.712589 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da843e22-9aa7-4320-98e7-c0cc38d484a8 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:30.724100 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1289/2585] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:41:30 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:41:30.728631 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c584062f-a2f4-46be-9a5c-da592c14efbd admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:30.741827 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c584062f-a2f4-46be-9a5c-da592c14efbd admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:30.747988 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6437a08b-8b87-43d0-8365-45c5bc364f3e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:30.748650 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6437a08b-8b87-43d0-8365-45c5bc364f3e admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:30.748934 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6437a08b-8b87-43d0-8365-45c5bc364f3e admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:30.749209 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6437a08b-8b87-43d0-8365-45c5bc364f3e admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:30.749963 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6437a08b-8b87-43d0-8365-45c5bc364f3e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:30.753092 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6437a08b-8b87-43d0-8365-45c5bc364f3e admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:30.764347 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1297/2586] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:41:30 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 29 21:41:30.769261 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da843e22-9aa7-4320-98e7-c0cc38d484a8 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:30.780083 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da843e22-9aa7-4320-98e7-c0cc38d484a8 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:30.785659 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7e97930-2d15-4841-a93b-ea7d2f7a5b17 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:30.786543 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e97930-2d15-4841-a93b-ea7d2f7a5b17 admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:30.786985 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e97930-2d15-4841-a93b-ea7d2f7a5b17 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:30.787447 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e97930-2d15-4841-a93b-ea7d2f7a5b17 admin admin] PATH_INFO: `/v3/projects/6223cf2c0b344bf5b2c44b8bbce5c30d/users/d12d33b35ebe474fa2da2e39e2d63f38/roles/02e242d5520740d4b77a5f4008c7eb57` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:30.807598 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7e97930-2d15-4841-a93b-ea7d2f7a5b17 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6223cf2c0b344bf5b2c44b8bbce5c30d, user_id=d12d33b35ebe474fa2da2e39e2d63f38, role_id=02e242d5520740d4b77a5f4008c7eb57)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:30.810779 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7e97930-2d15-4841-a93b-ea7d2f7a5b17 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:30.829326 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1290/2587] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:41:30 2022] PUT /identity/v3/projects/6223cf2c0b344bf5b2c44b8bbce5c30d/users/d12d33b35ebe474fa2da2e39e2d63f38/roles/02e242d5520740d4b77a5f4008c7eb57 => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:41:30.840606 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df98324c-5838-40a6-9375-62ee59a56192 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:30.840899 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df98324c-5838-40a6-9375-62ee59a56192 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:30.841211 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df98324c-5838-40a6-9375-62ee59a56192 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:30.873829 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-df98324c-5838-40a6-9375-62ee59a56192 None None] MFA Rules not processed for user `d12d33b35ebe474fa2da2e39e2d63f38`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:41:30.875218 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-df98324c-5838-40a6-9375-62ee59a56192 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:30.957370 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1298/2588] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:41:30 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:41:30.961683 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7e97930-2d15-4841-a93b-ea7d2f7a5b17 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:30.978886 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=98,keystone:DELETE=1,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:41:31.037290 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7e97930-2d15-4841-a93b-ea7d2f7a5b17 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:31.044643 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24f95e72-2635-4ed3-b06a-f3f92f8abdd7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:31.045183 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f95e72-2635-4ed3-b06a-f3f92f8abdd7 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:31.045767 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f95e72-2635-4ed3-b06a-f3f92f8abdd7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:31.046201 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f95e72-2635-4ed3-b06a-f3f92f8abdd7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:31.047413 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24f95e72-2635-4ed3-b06a-f3f92f8abdd7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:31.049926 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24f95e72-2635-4ed3-b06a-f3f92f8abdd7 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:31.056439 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df98324c-5838-40a6-9375-62ee59a56192 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:31.057964 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-df98324c-5838-40a6-9375-62ee59a56192 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:31.081775 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df98324c-5838-40a6-9375-62ee59a56192 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:31.087776 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77c5b046-4f87-4a00-9742-233e05a8f40d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:31.088849 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1291/2589] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:41:30 2022] POST /identity/v3/projects => generated 379 bytes in 128 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:41:31.092855 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24f95e72-2635-4ed3-b06a-f3f92f8abdd7 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:31.096881 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77c5b046-4f87-4a00-9742-233e05a8f40d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:31.097171 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77c5b046-4f87-4a00-9742-233e05a8f40d admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:31.097452 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77c5b046-4f87-4a00-9742-233e05a8f40d admin admin] PATH_INFO: `/v3/projects/809fa9baaebb4ac79d20d50288e41804` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:31.099773 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77c5b046-4f87-4a00-9742-233e05a8f40d admin admin] RBAC: Authorizing `identity:delete_project(project_id=809fa9baaebb4ac79d20d50288e41804)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:31.101938 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77c5b046-4f87-4a00-9742-233e05a8f40d admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:31.153771 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-77c5b046-4f87-4a00-9742-233e05a8f40d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '809fa9baaebb4ac79d20d50288e41804', 'request_id': 'req-77c5b046-4f87-4a00-9742-233e05a8f40d'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:31.160723 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-77c5b046-4f87-4a00-9742-233e05a8f40d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '809fa9baaebb4ac79d20d50288e41804', 'request_id': 'req-77c5b046-4f87-4a00-9742-233e05a8f40d'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:31.162032 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24f95e72-2635-4ed3-b06a-f3f92f8abdd7 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:31.165835 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=91,keystone:INSERT=3,keystone:DELETE=6 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:41:31.167588 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-77c5b046-4f87-4a00-9742-233e05a8f40d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '809fa9baaebb4ac79d20d50288e41804', 'request_id': 'req-77c5b046-4f87-4a00-9742-233e05a8f40d'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:31.182094 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-77c5b046-4f87-4a00-9742-233e05a8f40d admin admin] The token cache is being invalidate because project 809fa9baaebb4ac79d20d50288e41804 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:41:31.182779 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-77c5b046-4f87-4a00-9742-233e05a8f40d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:31.183430 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-77c5b046-4f87-4a00-9742-233e05a8f40d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:31.184979 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1299/2590] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:41:31 2022] DELETE /identity/v3/projects/809fa9baaebb4ac79d20d50288e41804 => generated 0 bytes in 129 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 21:41:31.189590 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77c5b046-4f87-4a00-9742-233e05a8f40d admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:31.191066 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-77c5b046-4f87-4a00-9742-233e05a8f40d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:31.226607 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-286ce642-a3f7-48ee-b781-0885aa140803 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:31.227141 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-286ce642-a3f7-48ee-b781-0885aa140803 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:31.227416 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-286ce642-a3f7-48ee-b781-0885aa140803 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:31.227686 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-286ce642-a3f7-48ee-b781-0885aa140803 admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:31.228550 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-286ce642-a3f7-48ee-b781-0885aa140803 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:31.230418 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-286ce642-a3f7-48ee-b781-0885aa140803 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:31.256048 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1292/2591] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:41:31 2022] POST /identity/v3/users => generated 296 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 21:41:31.257272 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-286ce642-a3f7-48ee-b781-0885aa140803 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:31.259677 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-286ce642-a3f7-48ee-b781-0885aa140803 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:31.262032 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77c5b046-4f87-4a00-9742-233e05a8f40d admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:31.267260 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6705da23-ca9f-4d0b-a19d-3cc9ef46ccb4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:31.267901 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6705da23-ca9f-4d0b-a19d-3cc9ef46ccb4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:31.268454 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6705da23-ca9f-4d0b-a19d-3cc9ef46ccb4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:31.268669 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6705da23-ca9f-4d0b-a19d-3cc9ef46ccb4 admin admin] PATH_INFO: `/v3/users/4f5c4879949a49d584c4fa3c0cce6925` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:31.269874 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6705da23-ca9f-4d0b-a19d-3cc9ef46ccb4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4f5c4879949a49d584c4fa3c0cce6925)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:31.271793 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6705da23-ca9f-4d0b-a19d-3cc9ef46ccb4 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:31.272426 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-286ce642-a3f7-48ee-b781-0885aa140803 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:31.278389 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3027cef-fa90-4145-b414-234c0c306c7a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:31.278990 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3027cef-fa90-4145-b414-234c0c306c7a admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:31.279269 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3027cef-fa90-4145-b414-234c0c306c7a admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:31.279538 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3027cef-fa90-4145-b414-234c0c306c7a admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:31.280171 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3027cef-fa90-4145-b414-234c0c306c7a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:31.281779 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3027cef-fa90-4145-b414-234c0c306c7a admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:31.292482 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1293/2592] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:41:31 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:41:31.296751 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3027cef-fa90-4145-b414-234c0c306c7a admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:31.308114 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3027cef-fa90-4145-b414-234c0c306c7a admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:31.312893 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9d6895c-a9e6-4603-bb9c-8bd49e78fd0f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:31.314932 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9d6895c-a9e6-4603-bb9c-8bd49e78fd0f admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:31.314932 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9d6895c-a9e6-4603-bb9c-8bd49e78fd0f admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:31.314932 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9d6895c-a9e6-4603-bb9c-8bd49e78fd0f admin admin] PATH_INFO: `/v3/projects/f94c5424f9524a8c9ed135d3de03544c/users/eedf91d1cf214273bda84e51602062e2/roles/a6e562cdc96c4be691d638990f8403fd` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:31.333709 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9d6895c-a9e6-4603-bb9c-8bd49e78fd0f admin admin] RBAC: Authorizing `identity:create_grant(project_id=f94c5424f9524a8c9ed135d3de03544c, user_id=eedf91d1cf214273bda84e51602062e2, role_id=a6e562cdc96c4be691d638990f8403fd)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:31.335982 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-6705da23-ca9f-4d0b-a19d-3cc9ef46ccb4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4f5c4879949a49d584c4fa3c0cce6925'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:31.336551 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9d6895c-a9e6-4603-bb9c-8bd49e78fd0f admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:31.346284 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:41:31.351410 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-6705da23-ca9f-4d0b-a19d-3cc9ef46ccb4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4f5c4879949a49d584c4fa3c0cce6925'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:31.352125 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-6705da23-ca9f-4d0b-a19d-3cc9ef46ccb4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4f5c4879949a49d584c4fa3c0cce6925'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:31.352775 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-6705da23-ca9f-4d0b-a19d-3cc9ef46ccb4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4f5c4879949a49d584c4fa3c0cce6925'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:31.354927 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1294/2593] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:41:31 2022] PUT /identity/v3/projects/f94c5424f9524a8c9ed135d3de03544c/users/eedf91d1cf214273bda84e51602062e2/roles/a6e562cdc96c4be691d638990f8403fd => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 21:41:31.357005 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9d6895c-a9e6-4603-bb9c-8bd49e78fd0f admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:31.359324 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-d9d6895c-a9e6-4603-bb9c-8bd49e78fd0f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:31.363978 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-6705da23-ca9f-4d0b-a19d-3cc9ef46ccb4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4f5c4879949a49d584c4fa3c0cce6925'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:31.380087 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1300/2594] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:41:31 2022] DELETE /identity/v3/users/4f5c4879949a49d584c4fa3c0cce6925 => generated 0 bytes in 191 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 21:41:31.434451 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9d6895c-a9e6-4603-bb9c-8bd49e78fd0f admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:31.505748 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-076a0c57-dbfd-4aa8-a173-e78c35c780fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:31.506795 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-076a0c57-dbfd-4aa8-a173-e78c35c780fb admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:31.507353 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-076a0c57-dbfd-4aa8-a173-e78c35c780fb admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:31.507852 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-076a0c57-dbfd-4aa8-a173-e78c35c780fb admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:31.509080 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-076a0c57-dbfd-4aa8-a173-e78c35c780fb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:31.511545 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-076a0c57-dbfd-4aa8-a173-e78c35c780fb admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:31.523900 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1295/2595] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:41:31 2022] GET /identity/v3/roles => generated 1420 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 29 21:41:31.528031 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6705da23-ca9f-4d0b-a19d-3cc9ef46ccb4 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:31.543025 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6705da23-ca9f-4d0b-a19d-3cc9ef46ccb4 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:31.549465 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00d1bc01-16f0-41c6-8dda-851d97056ca9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:31.549883 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00d1bc01-16f0-41c6-8dda-851d97056ca9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:31.550266 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00d1bc01-16f0-41c6-8dda-851d97056ca9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:31.550742 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00d1bc01-16f0-41c6-8dda-851d97056ca9 admin admin] PATH_INFO: `/v3/projects/f94c5424f9524a8c9ed135d3de03544c/users/eedf91d1cf214273bda84e51602062e2/roles/a6e562cdc96c4be691d638990f8403fd` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:31.553463 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00d1bc01-16f0-41c6-8dda-851d97056ca9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f94c5424f9524a8c9ed135d3de03544c, user_id=eedf91d1cf214273bda84e51602062e2, role_id=a6e562cdc96c4be691d638990f8403fd)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:31.557173 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00d1bc01-16f0-41c6-8dda-851d97056ca9 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:31.570954 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1301/2596] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:41:31 2022] PUT /identity/v3/projects/f94c5424f9524a8c9ed135d3de03544c/users/eedf91d1cf214273bda84e51602062e2/roles/a6e562cdc96c4be691d638990f8403fd => generated 0 bytes in 43 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 21:41:31.583396 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1bfaa70-438d-4da0-9338-f10cbb4dc0cb None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:31.583779 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1bfaa70-438d-4da0-9338-f10cbb4dc0cb None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:31.584030 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1bfaa70-438d-4da0-9338-f10cbb4dc0cb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:31.619682 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-f1bfaa70-438d-4da0-9338-f10cbb4dc0cb None None] MFA Rules not processed for user `eedf91d1cf214273bda84e51602062e2`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:41:31.621022 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-f1bfaa70-438d-4da0-9338-f10cbb4dc0cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:31.685190 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00d1bc01-16f0-41c6-8dda-851d97056ca9 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:31.687940 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-00d1bc01-16f0-41c6-8dda-851d97056ca9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:31.721500 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1296/2597] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:41:31 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:41:31.729016 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1bfaa70-438d-4da0-9338-f10cbb4dc0cb None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:31.733469 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-f1bfaa70-438d-4da0-9338-f10cbb4dc0cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:31.767605 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=86,keystone:INSERT=3,keystone:DELETE=11 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:41:31.775364 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:41:31.775804 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00d1bc01-16f0-41c6-8dda-851d97056ca9 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:31.783190 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-759a26ac-8b36-4f34-b3c5-6e26830626fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:31.783925 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-759a26ac-8b36-4f34-b3c5-6e26830626fc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:31.784308 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-759a26ac-8b36-4f34-b3c5-6e26830626fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:31.784657 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-759a26ac-8b36-4f34-b3c5-6e26830626fc admin admin] PATH_INFO: `/v3/projects/d541a155be98406ead40fa199d8cca56` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:31.785948 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-759a26ac-8b36-4f34-b3c5-6e26830626fc admin admin] RBAC: Authorizing `identity:delete_project(project_id=d541a155be98406ead40fa199d8cca56)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:31.787928 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-759a26ac-8b36-4f34-b3c5-6e26830626fc admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:31.805738 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1bfaa70-438d-4da0-9338-f10cbb4dc0cb None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:31.809858 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e69f88bc-a1a4-4cdb-af7d-276ff1cd25aa service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dc226f7ddcb49ecacb51ce57fdfc11b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:31.810482 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e69f88bc-a1a4-4cdb-af7d-276ff1cd25aa service cinder] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:31.810768 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e69f88bc-a1a4-4cdb-af7d-276ff1cd25aa service cinder] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:31.811041 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e69f88bc-a1a4-4cdb-af7d-276ff1cd25aa service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:31.824013 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e69f88bc-a1a4-4cdb-af7d-276ff1cd25aa service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:31.825502 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e69f88bc-a1a4-4cdb-af7d-276ff1cd25aa service cinder] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:31.845957 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1297/2598] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:41:31 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 117 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 21:41:31.853008 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-759a26ac-8b36-4f34-b3c5-6e26830626fc admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd541a155be98406ead40fa199d8cca56', 'request_id': 'req-759a26ac-8b36-4f34-b3c5-6e26830626fc'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:31.871205 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-759a26ac-8b36-4f34-b3c5-6e26830626fc admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd541a155be98406ead40fa199d8cca56', 'request_id': 'req-759a26ac-8b36-4f34-b3c5-6e26830626fc'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:31.872957 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-759a26ac-8b36-4f34-b3c5-6e26830626fc admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd541a155be98406ead40fa199d8cca56', 'request_id': 'req-759a26ac-8b36-4f34-b3c5-6e26830626fc'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:31.899685 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-759a26ac-8b36-4f34-b3c5-6e26830626fc admin admin] The token cache is being invalidate because project d541a155be98406ead40fa199d8cca56 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:41:31.902123 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-759a26ac-8b36-4f34-b3c5-6e26830626fc admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:31.902123 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-759a26ac-8b36-4f34-b3c5-6e26830626fc admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:31.903052 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1302/2599] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:41:31 2022] DELETE /identity/v3/projects/d541a155be98406ead40fa199d8cca56 => generated 0 bytes in 219 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 21:41:31.908894 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6641fc3b-add4-4da3-9c7b-f9ca5a827cc7 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:31.909252 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6641fc3b-add4-4da3-9c7b-f9ca5a827cc7 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:31.909597 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6641fc3b-add4-4da3-9c7b-f9ca5a827cc7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:31.949910 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-6641fc3b-add4-4da3-9c7b-f9ca5a827cc7 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:41:31.951160 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-6641fc3b-add4-4da3-9c7b-f9ca5a827cc7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:31.963204 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-759a26ac-8b36-4f34-b3c5-6e26830626fc admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:31.964746 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-759a26ac-8b36-4f34-b3c5-6e26830626fc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:32.026127 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1298/2600] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:41:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:41:32.035988 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb6db5be-c128-4db8-b1c3-2644d58bd024 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:32.036167 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb6db5be-c128-4db8-b1c3-2644d58bd024 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:32.036327 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb6db5be-c128-4db8-b1c3-2644d58bd024 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:32.037209 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-759a26ac-8b36-4f34-b3c5-6e26830626fc admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:32.042059 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-891d0938-d64e-44c9-932e-66c0ac51f81e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dc226f7ddcb49ecacb51ce57fdfc11b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:32.042703 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-891d0938-d64e-44c9-932e-66c0ac51f81e service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:32.043028 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-891d0938-d64e-44c9-932e-66c0ac51f81e service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:32.043232 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-891d0938-d64e-44c9-932e-66c0ac51f81e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:32.045183 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-891d0938-d64e-44c9-932e-66c0ac51f81e service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:32.057499 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-eb6db5be-c128-4db8-b1c3-2644d58bd024 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:41:32.058764 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-eb6db5be-c128-4db8-b1c3-2644d58bd024 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:32.067089 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1299/2601] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:41:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 21:41:32.072215 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb6db5be-c128-4db8-b1c3-2644d58bd024 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:32.082684 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb6db5be-c128-4db8-b1c3-2644d58bd024 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:32.087449 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-863211b1-869e-4696-9158-7e7adc4cfa6e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:32.088059 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-863211b1-869e-4696-9158-7e7adc4cfa6e admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:32.088376 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-863211b1-869e-4696-9158-7e7adc4cfa6e admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:32.088652 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-863211b1-869e-4696-9158-7e7adc4cfa6e admin admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:32.089307 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-863211b1-869e-4696-9158-7e7adc4cfa6e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:41:32.089648 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-863211b1-869e-4696-9158-7e7adc4cfa6e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:32.091350 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-863211b1-869e-4696-9158-7e7adc4cfa6e admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:32.104090 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1300/2602] 10.222.0.90 () {58 vars in 1153 bytes} [Mon Aug 29 21:41:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:41:32.108374 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-863211b1-869e-4696-9158-7e7adc4cfa6e admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:32.113936 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-891d0938-d64e-44c9-932e-66c0ac51f81e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:32.115385 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-891d0938-d64e-44c9-932e-66c0ac51f81e service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:32.119369 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-863211b1-869e-4696-9158-7e7adc4cfa6e admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:32.124461 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb597284-34e3-4b31-a958-edc6c2cf7340 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:32.124617 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1303/2603] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:41:31 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 162 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:41:32.127528 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb597284-34e3-4b31-a958-edc6c2cf7340 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:32.127817 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb597284-34e3-4b31-a958-edc6c2cf7340 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:32.128095 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb597284-34e3-4b31-a958-edc6c2cf7340 admin admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:32.128736 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb597284-34e3-4b31-a958-edc6c2cf7340 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:32.130676 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb597284-34e3-4b31-a958-edc6c2cf7340 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:32.161034 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1301/2604] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:41:32 2022] POST /identity/v3/projects => generated 373 bytes in 53 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 21:41:32.165639 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-891d0938-d64e-44c9-932e-66c0ac51f81e service cinder] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:32.176751 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-891d0938-d64e-44c9-932e-66c0ac51f81e service cinder] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:32.216905 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:41:32.240837 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cd87ac3-f2f2-46e7-91ea-2581021b63f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:32.241464 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cd87ac3-f2f2-46e7-91ea-2581021b63f5 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:32.241746 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cd87ac3-f2f2-46e7-91ea-2581021b63f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:32.242068 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cd87ac3-f2f2-46e7-91ea-2581021b63f5 admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:32.242696 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cd87ac3-f2f2-46e7-91ea-2581021b63f5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:32.244671 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cd87ac3-f2f2-46e7-91ea-2581021b63f5 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:32.268585 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1304/2605] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:41:32 2022] POST /identity/v3/users => generated 287 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:41:32.273069 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb597284-34e3-4b31-a958-edc6c2cf7340 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:32.285222 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb597284-34e3-4b31-a958-edc6c2cf7340 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:32.290872 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9ae2b90-306a-49aa-adca-0c9d6878add6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:32.291684 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9ae2b90-306a-49aa-adca-0c9d6878add6 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:32.291973 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9ae2b90-306a-49aa-adca-0c9d6878add6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:32.292275 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9ae2b90-306a-49aa-adca-0c9d6878add6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:32.293089 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9ae2b90-306a-49aa-adca-0c9d6878add6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:32.294862 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9ae2b90-306a-49aa-adca-0c9d6878add6 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:32.305501 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1302/2606] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:41:32 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:41:32.309709 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cd87ac3-f2f2-46e7-91ea-2581021b63f5 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:32.320586 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cd87ac3-f2f2-46e7-91ea-2581021b63f5 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:32.325497 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9510b4b6-9098-4a82-91af-1f4ddee1e059 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:32.326153 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9510b4b6-9098-4a82-91af-1f4ddee1e059 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:32.326459 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9510b4b6-9098-4a82-91af-1f4ddee1e059 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:32.326733 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9510b4b6-9098-4a82-91af-1f4ddee1e059 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:32.327631 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9510b4b6-9098-4a82-91af-1f4ddee1e059 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:32.329825 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9510b4b6-9098-4a82-91af-1f4ddee1e059 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:32.341173 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1305/2607] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:41:32 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 29 21:41:32.346854 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9ae2b90-306a-49aa-adca-0c9d6878add6 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:32.358158 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9ae2b90-306a-49aa-adca-0c9d6878add6 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:32.363887 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e829cb9-e4ec-4371-998d-dbfe6d46912d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:32.364549 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e829cb9-e4ec-4371-998d-dbfe6d46912d admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:32.364834 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e829cb9-e4ec-4371-998d-dbfe6d46912d admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:32.365135 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e829cb9-e4ec-4371-998d-dbfe6d46912d admin admin] PATH_INFO: `/v3/projects/aa72e265b0d244279a94d88db75b4e90/users/40a007fc85874fdba5d1b2ab8cb3d7a5/roles/02e242d5520740d4b77a5f4008c7eb57` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:32.380914 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e829cb9-e4ec-4371-998d-dbfe6d46912d admin admin] RBAC: Authorizing `identity:create_grant(project_id=aa72e265b0d244279a94d88db75b4e90, user_id=40a007fc85874fdba5d1b2ab8cb3d7a5, role_id=02e242d5520740d4b77a5f4008c7eb57)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:32.383619 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e829cb9-e4ec-4371-998d-dbfe6d46912d admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:32.399003 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1303/2608] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:41:32 2022] PUT /identity/v3/projects/aa72e265b0d244279a94d88db75b4e90/users/40a007fc85874fdba5d1b2ab8cb3d7a5/roles/02e242d5520740d4b77a5f4008c7eb57 => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 21:41:32.412335 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37c38b67-27df-40f0-845b-d1c79f8fe39b None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:32.412716 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37c38b67-27df-40f0-845b-d1c79f8fe39b None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:32.412998 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37c38b67-27df-40f0-845b-d1c79f8fe39b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:32.448309 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-37c38b67-27df-40f0-845b-d1c79f8fe39b None None] MFA Rules not processed for user `40a007fc85874fdba5d1b2ab8cb3d7a5`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:41:32.450625 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-37c38b67-27df-40f0-845b-d1c79f8fe39b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:32.525866 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1306/2609] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:41:32 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:41:32.539030 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e829cb9-e4ec-4371-998d-dbfe6d46912d admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:32.597043 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e829cb9-e4ec-4371-998d-dbfe6d46912d admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:32.602351 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f29e6424-ee77-434c-ae26-026ddb62a343 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dc226f7ddcb49ecacb51ce57fdfc11b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:32.603221 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f29e6424-ee77-434c-ae26-026ddb62a343 service cinder] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:32.603546 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f29e6424-ee77-434c-ae26-026ddb62a343 service cinder] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:32.603831 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f29e6424-ee77-434c-ae26-026ddb62a343 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:32.612490 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f29e6424-ee77-434c-ae26-026ddb62a343 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:32.613958 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f29e6424-ee77-434c-ae26-026ddb62a343 service cinder] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:32.617701 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:41:32.625598 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1304/2610] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:41:32 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:41:32.652748 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37c38b67-27df-40f0-845b-d1c79f8fe39b None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:32.722997 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37c38b67-27df-40f0-845b-d1c79f8fe39b None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:32.727756 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-351028db-529a-4964-9e63-4afb6acf4bfb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:32.728399 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-351028db-529a-4964-9e63-4afb6acf4bfb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:32.728683 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-351028db-529a-4964-9e63-4afb6acf4bfb admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:32.728956 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-351028db-529a-4964-9e63-4afb6acf4bfb admin admin] PATH_INFO: `/v3/users/40a007fc85874fdba5d1b2ab8cb3d7a5` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:32.730158 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-351028db-529a-4964-9e63-4afb6acf4bfb admin admin] RBAC: Authorizing `identity:delete_user(user_id=40a007fc85874fdba5d1b2ab8cb3d7a5)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:32.732117 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-351028db-529a-4964-9e63-4afb6acf4bfb admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:32.787071 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-351028db-529a-4964-9e63-4afb6acf4bfb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '40a007fc85874fdba5d1b2ab8cb3d7a5'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:32.789109 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=90,keystone:INSERT=3,keystone:DELETE=7 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:41:32.804627 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-351028db-529a-4964-9e63-4afb6acf4bfb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '40a007fc85874fdba5d1b2ab8cb3d7a5'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:32.805369 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-351028db-529a-4964-9e63-4afb6acf4bfb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '40a007fc85874fdba5d1b2ab8cb3d7a5'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:32.806011 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-351028db-529a-4964-9e63-4afb6acf4bfb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '40a007fc85874fdba5d1b2ab8cb3d7a5'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:32.814002 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-351028db-529a-4964-9e63-4afb6acf4bfb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '40a007fc85874fdba5d1b2ab8cb3d7a5'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:32.828727 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1307/2611] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:41:32 2022] DELETE /identity/v3/users/40a007fc85874fdba5d1b2ab8cb3d7a5 => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:41:32.833331 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3834773c-a56f-48ee-81a6-2f248285a55b None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:32.833624 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3834773c-a56f-48ee-81a6-2f248285a55b None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:32.833954 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3834773c-a56f-48ee-81a6-2f248285a55b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:32.854856 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-3834773c-a56f-48ee-81a6-2f248285a55b None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:41:32.856087 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-3834773c-a56f-48ee-81a6-2f248285a55b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:32.921149 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1305/2612] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:41:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:41:32.930910 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-351028db-529a-4964-9e63-4afb6acf4bfb admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:32.932355 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-351028db-529a-4964-9e63-4afb6acf4bfb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:32.990577 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-351028db-529a-4964-9e63-4afb6acf4bfb admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:32.994611 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb406659-d41c-4efd-b4e6-b6afe372bf6b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:32.995225 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb406659-d41c-4efd-b4e6-b6afe372bf6b service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:32.995592 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb406659-d41c-4efd-b4e6-b6afe372bf6b service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:32.995766 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb406659-d41c-4efd-b4e6-b6afe372bf6b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:33.005030 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb406659-d41c-4efd-b4e6-b6afe372bf6b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:33.006511 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb406659-d41c-4efd-b4e6-b6afe372bf6b service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:33.015732 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1308/2613] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:41:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:41:33.324443 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3834773c-a56f-48ee-81a6-2f248285a55b None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:33.325971 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-3834773c-a56f-48ee-81a6-2f248285a55b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:33.340357 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3834773c-a56f-48ee-81a6-2f248285a55b None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:33.344652 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e6fb067-a4f4-4099-b096-7343cc0aa6a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:33.345293 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e6fb067-a4f4-4099-b096-7343cc0aa6a1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:33.345571 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e6fb067-a4f4-4099-b096-7343cc0aa6a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:33.345875 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e6fb067-a4f4-4099-b096-7343cc0aa6a1 admin admin] PATH_INFO: `/v3/projects/aa72e265b0d244279a94d88db75b4e90` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:33.347178 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e6fb067-a4f4-4099-b096-7343cc0aa6a1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=aa72e265b0d244279a94d88db75b4e90)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:33.349069 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e6fb067-a4f4-4099-b096-7343cc0aa6a1 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:33.398738 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-9e6fb067-a4f4-4099-b096-7343cc0aa6a1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'aa72e265b0d244279a94d88db75b4e90', 'request_id': 'req-9e6fb067-a4f4-4099-b096-7343cc0aa6a1'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:33.404155 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-9e6fb067-a4f4-4099-b096-7343cc0aa6a1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'aa72e265b0d244279a94d88db75b4e90', 'request_id': 'req-9e6fb067-a4f4-4099-b096-7343cc0aa6a1'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:33.411757 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-9e6fb067-a4f4-4099-b096-7343cc0aa6a1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'aa72e265b0d244279a94d88db75b4e90', 'request_id': 'req-9e6fb067-a4f4-4099-b096-7343cc0aa6a1'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:33.424465 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-9e6fb067-a4f4-4099-b096-7343cc0aa6a1 admin admin] The token cache is being invalidate because project aa72e265b0d244279a94d88db75b4e90 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81434) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:41:33.424776 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-9e6fb067-a4f4-4099-b096-7343cc0aa6a1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:33.425423 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-9e6fb067-a4f4-4099-b096-7343cc0aa6a1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:33.426965 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1306/2614] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:41:33 2022] DELETE /identity/v3/projects/aa72e265b0d244279a94d88db75b4e90 => generated 0 bytes in 103 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:41:34.799239 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb406659-d41c-4efd-b4e6-b6afe372bf6b service neutron] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:34.800672 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-fb406659-d41c-4efd-b4e6-b6afe372bf6b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:34.814013 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb406659-d41c-4efd-b4e6-b6afe372bf6b service neutron] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:34.875367 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c028ed9-74e7-45d1-8867-3e3c709f570b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:34.876021 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c028ed9-74e7-45d1-8867-3e3c709f570b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:34.876297 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c028ed9-74e7-45d1-8867-3e3c709f570b admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:34.876574 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c028ed9-74e7-45d1-8867-3e3c709f570b admin admin] PATH_INFO: `/v3/users/d12d33b35ebe474fa2da2e39e2d63f38` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:34.877734 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c028ed9-74e7-45d1-8867-3e3c709f570b admin admin] RBAC: Authorizing `identity:delete_user(user_id=d12d33b35ebe474fa2da2e39e2d63f38)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:34.879701 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c028ed9-74e7-45d1-8867-3e3c709f570b admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:34.928652 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-1c028ed9-74e7-45d1-8867-3e3c709f570b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd12d33b35ebe474fa2da2e39e2d63f38'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:34.941260 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-1c028ed9-74e7-45d1-8867-3e3c709f570b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd12d33b35ebe474fa2da2e39e2d63f38'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:34.942008 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-1c028ed9-74e7-45d1-8867-3e3c709f570b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd12d33b35ebe474fa2da2e39e2d63f38'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:34.942701 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-1c028ed9-74e7-45d1-8867-3e3c709f570b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd12d33b35ebe474fa2da2e39e2d63f38'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:34.948105 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:41:34.950189 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-1c028ed9-74e7-45d1-8867-3e3c709f570b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd12d33b35ebe474fa2da2e39e2d63f38'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:34.965846 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1309/2615] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:41:34 2022] DELETE /identity/v3/users/d12d33b35ebe474fa2da2e39e2d63f38 => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:41:34.970860 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84981cc7-c454-42f3-b0da-b8ba365e3fd1 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:34.971492 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84981cc7-c454-42f3-b0da-b8ba365e3fd1 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:34.971606 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84981cc7-c454-42f3-b0da-b8ba365e3fd1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:34.992934 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-84981cc7-c454-42f3-b0da-b8ba365e3fd1 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:41:34.994284 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-84981cc7-c454-42f3-b0da-b8ba365e3fd1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:35.073184 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1307/2616] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:41:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:41:35.080472 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c028ed9-74e7-45d1-8867-3e3c709f570b admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:35.082507 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-1c028ed9-74e7-45d1-8867-3e3c709f570b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:35.145702 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c028ed9-74e7-45d1-8867-3e3c709f570b admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:35.149393 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee996235-24a1-49bc-8470-9a9a54867b25 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:35.150037 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee996235-24a1-49bc-8470-9a9a54867b25 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:35.150312 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee996235-24a1-49bc-8470-9a9a54867b25 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:35.150582 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee996235-24a1-49bc-8470-9a9a54867b25 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:35.160318 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee996235-24a1-49bc-8470-9a9a54867b25 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:35.161717 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee996235-24a1-49bc-8470-9a9a54867b25 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:35.171044 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1310/2617] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:41:35 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:41:35.466051 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84981cc7-c454-42f3-b0da-b8ba365e3fd1 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:35.467569 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-84981cc7-c454-42f3-b0da-b8ba365e3fd1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:35.481082 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84981cc7-c454-42f3-b0da-b8ba365e3fd1 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:35.486731 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fc705fa-d9fe-4ab1-885b-5518fcdfc40b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:35.487472 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fc705fa-d9fe-4ab1-885b-5518fcdfc40b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:35.488353 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fc705fa-d9fe-4ab1-885b-5518fcdfc40b admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:35.488725 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fc705fa-d9fe-4ab1-885b-5518fcdfc40b admin admin] PATH_INFO: `/v3/projects/6223cf2c0b344bf5b2c44b8bbce5c30d` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:35.489988 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fc705fa-d9fe-4ab1-885b-5518fcdfc40b admin admin] RBAC: Authorizing `identity:delete_project(project_id=6223cf2c0b344bf5b2c44b8bbce5c30d)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:35.491962 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fc705fa-d9fe-4ab1-885b-5518fcdfc40b admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:35.512792 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:41:35.546863 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-1fc705fa-d9fe-4ab1-885b-5518fcdfc40b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6223cf2c0b344bf5b2c44b8bbce5c30d', 'request_id': 'req-1fc705fa-d9fe-4ab1-885b-5518fcdfc40b'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:35.553113 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-1fc705fa-d9fe-4ab1-885b-5518fcdfc40b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6223cf2c0b344bf5b2c44b8bbce5c30d', 'request_id': 'req-1fc705fa-d9fe-4ab1-885b-5518fcdfc40b'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:35.559445 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-1fc705fa-d9fe-4ab1-885b-5518fcdfc40b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6223cf2c0b344bf5b2c44b8bbce5c30d', 'request_id': 'req-1fc705fa-d9fe-4ab1-885b-5518fcdfc40b'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:35.573386 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-1fc705fa-d9fe-4ab1-885b-5518fcdfc40b admin admin] The token cache is being invalidate because project 6223cf2c0b344bf5b2c44b8bbce5c30d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81434) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:41:35.573674 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-1fc705fa-d9fe-4ab1-885b-5518fcdfc40b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:35.574324 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-1fc705fa-d9fe-4ab1-885b-5518fcdfc40b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:35.575825 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1308/2618] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:41:35 2022] DELETE /identity/v3/projects/6223cf2c0b344bf5b2c44b8bbce5c30d => generated 0 bytes in 110 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:41:35.580773 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee996235-24a1-49bc-8470-9a9a54867b25 service neutron] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:35.590887 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee996235-24a1-49bc-8470-9a9a54867b25 service neutron] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:35.647581 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fc705fa-d9fe-4ab1-885b-5518fcdfc40b admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:35.648987 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-1fc705fa-d9fe-4ab1-885b-5518fcdfc40b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:35.654692 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35be84af-e57b-4d77-8e91-dcb39aed6f65 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:35.655347 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35be84af-e57b-4d77-8e91-dcb39aed6f65 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:35.655627 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35be84af-e57b-4d77-8e91-dcb39aed6f65 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:35.655905 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35be84af-e57b-4d77-8e91-dcb39aed6f65 admin admin] PATH_INFO: `/v3/users/eedf91d1cf214273bda84e51602062e2` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:35.657878 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35be84af-e57b-4d77-8e91-dcb39aed6f65 admin admin] RBAC: Authorizing `identity:delete_user(user_id=eedf91d1cf214273bda84e51602062e2)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:35.659718 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35be84af-e57b-4d77-8e91-dcb39aed6f65 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:35.716478 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-35be84af-e57b-4d77-8e91-dcb39aed6f65 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'eedf91d1cf214273bda84e51602062e2'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:35.731406 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-35be84af-e57b-4d77-8e91-dcb39aed6f65 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'eedf91d1cf214273bda84e51602062e2'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:35.732141 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-35be84af-e57b-4d77-8e91-dcb39aed6f65 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'eedf91d1cf214273bda84e51602062e2'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:35.732772 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-35be84af-e57b-4d77-8e91-dcb39aed6f65 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'eedf91d1cf214273bda84e51602062e2'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:35.740927 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-35be84af-e57b-4d77-8e91-dcb39aed6f65 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'eedf91d1cf214273bda84e51602062e2'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:35.752550 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fc705fa-d9fe-4ab1-885b-5518fcdfc40b admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:35.755604 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1311/2619] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:41:35 2022] DELETE /identity/v3/users/eedf91d1cf214273bda84e51602062e2 => generated 0 bytes in 175 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:41:35.758737 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e411f8eb-7d50-4c7f-b17c-4490c9d7201a service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b95fa07a7c0549dbaac49406e2639e21', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:35.759304 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e411f8eb-7d50-4c7f-b17c-4490c9d7201a service placement] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:35.759584 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e411f8eb-7d50-4c7f-b17c-4490c9d7201a service placement] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:35.759909 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e411f8eb-7d50-4c7f-b17c-4490c9d7201a service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:35.762092 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-e411f8eb-7d50-4c7f-b17c-4490c9d7201a service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:35.774886 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e411f8eb-7d50-4c7f-b17c-4490c9d7201a service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:35.776645 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e411f8eb-7d50-4c7f-b17c-4490c9d7201a service placement] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:35.788505 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1309/2620] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:41:35 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 141 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 21:41:36.043298 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35be84af-e57b-4d77-8e91-dcb39aed6f65 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:36.045032 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-35be84af-e57b-4d77-8e91-dcb39aed6f65 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:36.061242 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=87,keystone:INSERT=2,keystone:DELETE=11 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:41:36.120365 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35be84af-e57b-4d77-8e91-dcb39aed6f65 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:36.125038 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bdd2984-cf67-470d-87de-0c34ccdbec91 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:36.125944 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bdd2984-cf67-470d-87de-0c34ccdbec91 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:36.126713 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bdd2984-cf67-470d-87de-0c34ccdbec91 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:36.126996 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bdd2984-cf67-470d-87de-0c34ccdbec91 admin admin] PATH_INFO: `/v3/projects/f94c5424f9524a8c9ed135d3de03544c` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:36.128281 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bdd2984-cf67-470d-87de-0c34ccdbec91 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f94c5424f9524a8c9ed135d3de03544c)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:36.130287 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bdd2984-cf67-470d-87de-0c34ccdbec91 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:36.180360 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-4bdd2984-cf67-470d-87de-0c34ccdbec91 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f94c5424f9524a8c9ed135d3de03544c', 'request_id': 'req-4bdd2984-cf67-470d-87de-0c34ccdbec91'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:36.186803 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-4bdd2984-cf67-470d-87de-0c34ccdbec91 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f94c5424f9524a8c9ed135d3de03544c', 'request_id': 'req-4bdd2984-cf67-470d-87de-0c34ccdbec91'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:36.193513 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-4bdd2984-cf67-470d-87de-0c34ccdbec91 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f94c5424f9524a8c9ed135d3de03544c', 'request_id': 'req-4bdd2984-cf67-470d-87de-0c34ccdbec91'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:36.207337 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-4bdd2984-cf67-470d-87de-0c34ccdbec91 admin admin] The token cache is being invalidate because project f94c5424f9524a8c9ed135d3de03544c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:41:36.207628 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-4bdd2984-cf67-470d-87de-0c34ccdbec91 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:36.208258 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-4bdd2984-cf67-470d-87de-0c34ccdbec91 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:36.209766 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1312/2621] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:41:36 2022] DELETE /identity/v3/projects/f94c5424f9524a8c9ed135d3de03544c => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:41:36.214783 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d263af0-f56c-46b0-a46f-cd44e2dfb8df None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:36.215280 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d263af0-f56c-46b0-a46f-cd44e2dfb8df None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:36.215384 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d263af0-f56c-46b0-a46f-cd44e2dfb8df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:36.235866 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-6d263af0-f56c-46b0-a46f-cd44e2dfb8df None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:41:36.237179 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-6d263af0-f56c-46b0-a46f-cd44e2dfb8df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:36.297281 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1310/2622] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:41:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:41:36.305290 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17603908-7ae9-4c7e-a214-44ce30ac5c76 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:36.305610 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17603908-7ae9-4c7e-a214-44ce30ac5c76 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:36.305925 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17603908-7ae9-4c7e-a214-44ce30ac5c76 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:36.326261 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-17603908-7ae9-4c7e-a214-44ce30ac5c76 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:41:36.327669 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-17603908-7ae9-4c7e-a214-44ce30ac5c76 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:36.336824 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1313/2623] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:41:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:41:36.341267 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d263af0-f56c-46b0-a46f-cd44e2dfb8df None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:36.351094 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d263af0-f56c-46b0-a46f-cd44e2dfb8df None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:36.355953 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fee8ff9-296f-4b57-853c-3bfb455f0649 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:36.356673 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fee8ff9-296f-4b57-853c-3bfb455f0649 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:36.356960 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fee8ff9-296f-4b57-853c-3bfb455f0649 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:36.357235 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fee8ff9-296f-4b57-853c-3bfb455f0649 admin admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:36.357926 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fee8ff9-296f-4b57-853c-3bfb455f0649 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:41:36.358238 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fee8ff9-296f-4b57-853c-3bfb455f0649 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:36.359837 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fee8ff9-296f-4b57-853c-3bfb455f0649 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:36.373372 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1311/2624] 10.222.0.90 () {58 vars in 1153 bytes} [Mon Aug 29 21:41:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:41:36.378720 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17603908-7ae9-4c7e-a214-44ce30ac5c76 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:36.388571 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17603908-7ae9-4c7e-a214-44ce30ac5c76 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:36.394754 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f25431fd-ab6a-47f1-9ee1-c034b5d9380d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:36.395396 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f25431fd-ab6a-47f1-9ee1-c034b5d9380d admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:36.395675 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f25431fd-ab6a-47f1-9ee1-c034b5d9380d admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:36.395946 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f25431fd-ab6a-47f1-9ee1-c034b5d9380d admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:36.396589 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f25431fd-ab6a-47f1-9ee1-c034b5d9380d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:36.398565 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f25431fd-ab6a-47f1-9ee1-c034b5d9380d admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:36.431514 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1314/2625] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:41:36 2022] POST /identity/v3/projects => generated 387 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:41:36.432964 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fee8ff9-296f-4b57-853c-3bfb455f0649 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:36.439468 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:41:36.446610 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fee8ff9-296f-4b57-853c-3bfb455f0649 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:36.504757 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c998d3d-5cce-45b2-af6b-fbe5b676c1ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:36.505371 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c998d3d-5cce-45b2-af6b-fbe5b676c1ed admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:36.505648 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c998d3d-5cce-45b2-af6b-fbe5b676c1ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:36.505950 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c998d3d-5cce-45b2-af6b-fbe5b676c1ed admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:36.506669 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c998d3d-5cce-45b2-af6b-fbe5b676c1ed admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:36.508572 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c998d3d-5cce-45b2-af6b-fbe5b676c1ed admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:36.533337 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1312/2626] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:41:36 2022] POST /identity/v3/users => generated 294 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 21:41:36.537384 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f25431fd-ab6a-47f1-9ee1-c034b5d9380d admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:36.548880 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f25431fd-ab6a-47f1-9ee1-c034b5d9380d admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:36.553986 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8ce83eb-4682-4b58-9d1f-dbcee04f3945 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:36.554605 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8ce83eb-4682-4b58-9d1f-dbcee04f3945 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:36.554886 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8ce83eb-4682-4b58-9d1f-dbcee04f3945 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:36.555161 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8ce83eb-4682-4b58-9d1f-dbcee04f3945 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:36.555798 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8ce83eb-4682-4b58-9d1f-dbcee04f3945 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:36.557484 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8ce83eb-4682-4b58-9d1f-dbcee04f3945 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:36.568863 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1315/2627] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:41:36 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 29 21:41:36.572861 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c998d3d-5cce-45b2-af6b-fbe5b676c1ed admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:36.584151 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c998d3d-5cce-45b2-af6b-fbe5b676c1ed admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:36.588811 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbf0c78c-6346-4dd5-a875-5d5adce1e56f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:36.589416 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbf0c78c-6346-4dd5-a875-5d5adce1e56f admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:36.589699 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbf0c78c-6346-4dd5-a875-5d5adce1e56f admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:36.590013 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbf0c78c-6346-4dd5-a875-5d5adce1e56f admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:36.590654 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbf0c78c-6346-4dd5-a875-5d5adce1e56f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:36.592247 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbf0c78c-6346-4dd5-a875-5d5adce1e56f admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:36.604185 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1313/2628] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:41:36 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:41:36.608460 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8ce83eb-4682-4b58-9d1f-dbcee04f3945 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:36.620014 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8ce83eb-4682-4b58-9d1f-dbcee04f3945 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:36.624788 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d6f45c7-8ca1-4765-aa0d-159405a99966 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:36.625444 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d6f45c7-8ca1-4765-aa0d-159405a99966 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:36.626742 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d6f45c7-8ca1-4765-aa0d-159405a99966 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:36.627182 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d6f45c7-8ca1-4765-aa0d-159405a99966 admin admin] PATH_INFO: `/v3/projects/f3a84b8674d54fdeb63515902e40c9e9/users/9d43b13ace804beca480d5ca8f0c1164/roles/02e242d5520740d4b77a5f4008c7eb57` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:36.643335 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d6f45c7-8ca1-4765-aa0d-159405a99966 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f3a84b8674d54fdeb63515902e40c9e9, user_id=9d43b13ace804beca480d5ca8f0c1164, role_id=02e242d5520740d4b77a5f4008c7eb57)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:36.646477 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d6f45c7-8ca1-4765-aa0d-159405a99966 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:36.662999 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1316/2629] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:41:36 2022] PUT /identity/v3/projects/f3a84b8674d54fdeb63515902e40c9e9/users/9d43b13ace804beca480d5ca8f0c1164/roles/02e242d5520740d4b77a5f4008c7eb57 => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:41:36.677760 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-509a0dd3-f0cc-4c3f-bb84-8227401a8f16 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:36.678076 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-509a0dd3-f0cc-4c3f-bb84-8227401a8f16 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:36.678343 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-509a0dd3-f0cc-4c3f-bb84-8227401a8f16 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:36.708892 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-509a0dd3-f0cc-4c3f-bb84-8227401a8f16 None None] MFA Rules not processed for user `9d43b13ace804beca480d5ca8f0c1164`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:41:36.710199 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-509a0dd3-f0cc-4c3f-bb84-8227401a8f16 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:36.786486 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1314/2630] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:41:36 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:41:36.791022 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d6f45c7-8ca1-4765-aa0d-159405a99966 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:36.839297 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:41:36.871854 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d6f45c7-8ca1-4765-aa0d-159405a99966 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:36.878391 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef887adf-a933-4b86-ab21-8f4f11f4bb13 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:36.879672 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef887adf-a933-4b86-ab21-8f4f11f4bb13 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:36.879672 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef887adf-a933-4b86-ab21-8f4f11f4bb13 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:36.879672 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef887adf-a933-4b86-ab21-8f4f11f4bb13 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:36.880201 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef887adf-a933-4b86-ab21-8f4f11f4bb13 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:36.882177 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef887adf-a933-4b86-ab21-8f4f11f4bb13 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:36.917727 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1317/2631] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:41:36 2022] POST /identity/v3/projects => generated 387 bytes in 127 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:41:36.922109 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-509a0dd3-f0cc-4c3f-bb84-8227401a8f16 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:36.935016 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-509a0dd3-f0cc-4c3f-bb84-8227401a8f16 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:36.997628 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eca4ca97-86ca-47ee-809a-c26b98194193 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:36.998288 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eca4ca97-86ca-47ee-809a-c26b98194193 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:36.998576 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eca4ca97-86ca-47ee-809a-c26b98194193 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:36.998852 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eca4ca97-86ca-47ee-809a-c26b98194193 admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:36.999494 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eca4ca97-86ca-47ee-809a-c26b98194193 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:37.002192 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eca4ca97-86ca-47ee-809a-c26b98194193 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:37.019994 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:41:37.029217 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1315/2632] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:41:36 2022] POST /identity/v3/users => generated 300 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:41:37.030521 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef887adf-a933-4b86-ab21-8f4f11f4bb13 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:37.043164 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef887adf-a933-4b86-ab21-8f4f11f4bb13 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:37.048823 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69ebce29-a77f-46e9-b152-d064db185f0d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:37.049466 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ebce29-a77f-46e9-b152-d064db185f0d admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:37.049567 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ebce29-a77f-46e9-b152-d064db185f0d admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:37.049859 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ebce29-a77f-46e9-b152-d064db185f0d admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:37.050524 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69ebce29-a77f-46e9-b152-d064db185f0d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:37.052453 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69ebce29-a77f-46e9-b152-d064db185f0d admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:37.065916 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1318/2633] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:41:37 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:41:37.069917 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eca4ca97-86ca-47ee-809a-c26b98194193 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:37.080991 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eca4ca97-86ca-47ee-809a-c26b98194193 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:37.086183 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db0ba01d-baac-4cf6-b976-0cec332ef0d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:37.086855 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db0ba01d-baac-4cf6-b976-0cec332ef0d0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:37.087134 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db0ba01d-baac-4cf6-b976-0cec332ef0d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:37.087406 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db0ba01d-baac-4cf6-b976-0cec332ef0d0 admin admin] PATH_INFO: `/v3/projects/3511b0d75a7c44278781e6cfddc958c5/users/f38b3aee159a4cf3b0a684ea8c341abe/roles/a6e562cdc96c4be691d638990f8403fd` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:37.104087 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db0ba01d-baac-4cf6-b976-0cec332ef0d0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3511b0d75a7c44278781e6cfddc958c5, user_id=f38b3aee159a4cf3b0a684ea8c341abe, role_id=a6e562cdc96c4be691d638990f8403fd)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:37.106691 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db0ba01d-baac-4cf6-b976-0cec332ef0d0 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:37.120806 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1316/2634] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:41:37 2022] PUT /identity/v3/projects/3511b0d75a7c44278781e6cfddc958c5/users/f38b3aee159a4cf3b0a684ea8c341abe/roles/a6e562cdc96c4be691d638990f8403fd => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:41:37.124844 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69ebce29-a77f-46e9-b152-d064db185f0d admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:37.135867 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69ebce29-a77f-46e9-b152-d064db185f0d admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:37.203373 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0978ea10-0cc0-4092-bad8-12ce30760093 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:37.204038 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0978ea10-0cc0-4092-bad8-12ce30760093 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:37.204038 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0978ea10-0cc0-4092-bad8-12ce30760093 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:37.204638 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0978ea10-0cc0-4092-bad8-12ce30760093 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:37.205168 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0978ea10-0cc0-4092-bad8-12ce30760093 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:37.206858 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0978ea10-0cc0-4092-bad8-12ce30760093 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:37.220823 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1319/2635] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:41:37 2022] GET /identity/v3/roles => generated 1420 bytes in 93 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:41:37.223263 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db0ba01d-baac-4cf6-b976-0cec332ef0d0 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:37.234134 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db0ba01d-baac-4cf6-b976-0cec332ef0d0 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:37.239891 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3dfd2ce-cf98-439b-86b1-74aa45e7a6bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:37.240568 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3dfd2ce-cf98-439b-86b1-74aa45e7a6bb admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:37.240949 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3dfd2ce-cf98-439b-86b1-74aa45e7a6bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:37.241118 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3dfd2ce-cf98-439b-86b1-74aa45e7a6bb admin admin] PATH_INFO: `/v3/projects/3511b0d75a7c44278781e6cfddc958c5/users/f38b3aee159a4cf3b0a684ea8c341abe/roles/a6e562cdc96c4be691d638990f8403fd` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:37.243153 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3dfd2ce-cf98-439b-86b1-74aa45e7a6bb admin admin] RBAC: Authorizing `identity:create_grant(project_id=3511b0d75a7c44278781e6cfddc958c5, user_id=f38b3aee159a4cf3b0a684ea8c341abe, role_id=a6e562cdc96c4be691d638990f8403fd)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:37.245748 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3dfd2ce-cf98-439b-86b1-74aa45e7a6bb admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:37.258004 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1317/2636] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:41:37 2022] PUT /identity/v3/projects/3511b0d75a7c44278781e6cfddc958c5/users/f38b3aee159a4cf3b0a684ea8c341abe/roles/a6e562cdc96c4be691d638990f8403fd => generated 0 bytes in 36 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:41:37.269629 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95c63268-ccfa-41c5-9edb-acbfc1762386 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:37.269972 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95c63268-ccfa-41c5-9edb-acbfc1762386 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:37.270253 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95c63268-ccfa-41c5-9edb-acbfc1762386 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:37.300872 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-95c63268-ccfa-41c5-9edb-acbfc1762386 None None] MFA Rules not processed for user `f38b3aee159a4cf3b0a684ea8c341abe`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:41:37.302105 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-95c63268-ccfa-41c5-9edb-acbfc1762386 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:37.387514 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1320/2637] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:41:37 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:41:37.398612 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3dfd2ce-cf98-439b-86b1-74aa45e7a6bb admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:37.400542 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-d3dfd2ce-cf98-439b-86b1-74aa45e7a6bb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:37.463024 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3dfd2ce-cf98-439b-86b1-74aa45e7a6bb admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:37.467072 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3e5e56b-01d4-41d8-98a6-36787d12a6c0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dc226f7ddcb49ecacb51ce57fdfc11b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:37.467720 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e5e56b-01d4-41d8-98a6-36787d12a6c0 service cinder] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:37.468003 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e5e56b-01d4-41d8-98a6-36787d12a6c0 service cinder] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:37.468270 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e5e56b-01d4-41d8-98a6-36787d12a6c0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:37.478560 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3e5e56b-01d4-41d8-98a6-36787d12a6c0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:37.479994 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3e5e56b-01d4-41d8-98a6-36787d12a6c0 service cinder] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:37.490949 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1318/2638] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:41:37 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:41:38.075710 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95c63268-ccfa-41c5-9edb-acbfc1762386 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:38.111290 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:41:38.140516 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95c63268-ccfa-41c5-9edb-acbfc1762386 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:38.146257 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0cb4cd8-8276-4fbe-96d6-3b2c8a8d1996 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:38.146904 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0cb4cd8-8276-4fbe-96d6-3b2c8a8d1996 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:38.147222 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0cb4cd8-8276-4fbe-96d6-3b2c8a8d1996 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:38.147598 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0cb4cd8-8276-4fbe-96d6-3b2c8a8d1996 admin admin] PATH_INFO: `/v3/users/9d43b13ace804beca480d5ca8f0c1164` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:38.148878 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0cb4cd8-8276-4fbe-96d6-3b2c8a8d1996 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9d43b13ace804beca480d5ca8f0c1164)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:38.150884 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0cb4cd8-8276-4fbe-96d6-3b2c8a8d1996 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:38.200389 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-c0cb4cd8-8276-4fbe-96d6-3b2c8a8d1996 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9d43b13ace804beca480d5ca8f0c1164'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:38.214683 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-c0cb4cd8-8276-4fbe-96d6-3b2c8a8d1996 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9d43b13ace804beca480d5ca8f0c1164'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:38.215408 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-c0cb4cd8-8276-4fbe-96d6-3b2c8a8d1996 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9d43b13ace804beca480d5ca8f0c1164'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:38.216022 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-c0cb4cd8-8276-4fbe-96d6-3b2c8a8d1996 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9d43b13ace804beca480d5ca8f0c1164'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:38.224148 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-c0cb4cd8-8276-4fbe-96d6-3b2c8a8d1996 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9d43b13ace804beca480d5ca8f0c1164'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:38.237539 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1321/2639] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:41:38 2022] DELETE /identity/v3/users/9d43b13ace804beca480d5ca8f0c1164 => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:41:38.242395 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e6d0cc7-651e-4cf4-83c0-36d1232130f1 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:38.242868 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e6d0cc7-651e-4cf4-83c0-36d1232130f1 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:38.242968 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e6d0cc7-651e-4cf4-83c0-36d1232130f1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:38.262248 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-9e6d0cc7-651e-4cf4-83c0-36d1232130f1 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:41:38.263508 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-9e6d0cc7-651e-4cf4-83c0-36d1232130f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:38.323832 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1319/2640] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:41:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:41:38.334649 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0cb4cd8-8276-4fbe-96d6-3b2c8a8d1996 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:38.336154 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-c0cb4cd8-8276-4fbe-96d6-3b2c8a8d1996 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:38.401181 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0cb4cd8-8276-4fbe-96d6-3b2c8a8d1996 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:38.404771 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9583436-8771-4a96-9b4b-5679d0641a36 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:38.405378 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9583436-8771-4a96-9b4b-5679d0641a36 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:38.405660 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9583436-8771-4a96-9b4b-5679d0641a36 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:38.405978 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9583436-8771-4a96-9b4b-5679d0641a36 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:38.416235 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9583436-8771-4a96-9b4b-5679d0641a36 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:38.417724 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9583436-8771-4a96-9b4b-5679d0641a36 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:38.428226 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1322/2641] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:41:38 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:41:38.798674 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e6d0cc7-651e-4cf4-83c0-36d1232130f1 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:38.800391 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-9e6d0cc7-651e-4cf4-83c0-36d1232130f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:38.812859 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e6d0cc7-651e-4cf4-83c0-36d1232130f1 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:38.826518 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9837b68-ecf5-4416-8c4b-5c16ca788ccc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:38.827468 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9837b68-ecf5-4416-8c4b-5c16ca788ccc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:38.827748 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9837b68-ecf5-4416-8c4b-5c16ca788ccc admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:38.828020 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9837b68-ecf5-4416-8c4b-5c16ca788ccc admin admin] PATH_INFO: `/v3/projects/f3a84b8674d54fdeb63515902e40c9e9` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:38.829859 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9837b68-ecf5-4416-8c4b-5c16ca788ccc admin admin] RBAC: Authorizing `identity:delete_project(project_id=f3a84b8674d54fdeb63515902e40c9e9)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:38.832756 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9837b68-ecf5-4416-8c4b-5c16ca788ccc admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:38.903678 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-d9837b68-ecf5-4416-8c4b-5c16ca788ccc admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f3a84b8674d54fdeb63515902e40c9e9', 'request_id': 'req-d9837b68-ecf5-4416-8c4b-5c16ca788ccc'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:38.910567 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-d9837b68-ecf5-4416-8c4b-5c16ca788ccc admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f3a84b8674d54fdeb63515902e40c9e9', 'request_id': 'req-d9837b68-ecf5-4416-8c4b-5c16ca788ccc'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:38.913656 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=92,keystone:INSERT=2,keystone:DELETE=6 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:41:38.918662 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-d9837b68-ecf5-4416-8c4b-5c16ca788ccc admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f3a84b8674d54fdeb63515902e40c9e9', 'request_id': 'req-d9837b68-ecf5-4416-8c4b-5c16ca788ccc'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:38.934018 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-d9837b68-ecf5-4416-8c4b-5c16ca788ccc admin admin] The token cache is being invalidate because project f3a84b8674d54fdeb63515902e40c9e9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81434) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:41:38.934310 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-d9837b68-ecf5-4416-8c4b-5c16ca788ccc admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:38.934947 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-d9837b68-ecf5-4416-8c4b-5c16ca788ccc admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:38.936553 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1320/2642] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:41:38 2022] DELETE /identity/v3/projects/f3a84b8674d54fdeb63515902e40c9e9 => generated 0 bytes in 138 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:41:38.941639 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9583436-8771-4a96-9b4b-5679d0641a36 service neutron] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:38.951752 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9583436-8771-4a96-9b4b-5679d0641a36 service neutron] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:39.008511 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:41:39.016598 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b9b349c-cd63-4b8a-9261-e855a5cc7cda admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:39.018460 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b9b349c-cd63-4b8a-9261-e855a5cc7cda admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:39.018781 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b9b349c-cd63-4b8a-9261-e855a5cc7cda admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:39.019135 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b9b349c-cd63-4b8a-9261-e855a5cc7cda admin admin] PATH_INFO: `/v3/users/f38b3aee159a4cf3b0a684ea8c341abe` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:39.020653 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b9b349c-cd63-4b8a-9261-e855a5cc7cda admin admin] RBAC: Authorizing `identity:delete_user(user_id=f38b3aee159a4cf3b0a684ea8c341abe)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:39.022941 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b9b349c-cd63-4b8a-9261-e855a5cc7cda admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:39.081846 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-1b9b349c-cd63-4b8a-9261-e855a5cc7cda admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f38b3aee159a4cf3b0a684ea8c341abe'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:39.096294 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-1b9b349c-cd63-4b8a-9261-e855a5cc7cda admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f38b3aee159a4cf3b0a684ea8c341abe'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:39.097407 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-1b9b349c-cd63-4b8a-9261-e855a5cc7cda admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f38b3aee159a4cf3b0a684ea8c341abe'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:39.098039 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-1b9b349c-cd63-4b8a-9261-e855a5cc7cda admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f38b3aee159a4cf3b0a684ea8c341abe'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:39.105288 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-1b9b349c-cd63-4b8a-9261-e855a5cc7cda admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f38b3aee159a4cf3b0a684ea8c341abe'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:39.122084 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1323/2643] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:41:38 2022] DELETE /identity/v3/users/f38b3aee159a4cf3b0a684ea8c341abe => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:41:39.418886 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9837b68-ecf5-4416-8c4b-5c16ca788ccc admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:39.420472 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-d9837b68-ecf5-4416-8c4b-5c16ca788ccc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:39.437123 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9837b68-ecf5-4416-8c4b-5c16ca788ccc admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:39.507459 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-020d9272-8989-4455-b587-7f727b2c3407 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:39.508200 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-020d9272-8989-4455-b587-7f727b2c3407 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:39.508550 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-020d9272-8989-4455-b587-7f727b2c3407 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:39.508895 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-020d9272-8989-4455-b587-7f727b2c3407 admin admin] PATH_INFO: `/v3/projects/3511b0d75a7c44278781e6cfddc958c5` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:39.510293 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-020d9272-8989-4455-b587-7f727b2c3407 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3511b0d75a7c44278781e6cfddc958c5)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:39.512511 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-020d9272-8989-4455-b587-7f727b2c3407 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:39.567182 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-020d9272-8989-4455-b587-7f727b2c3407 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3511b0d75a7c44278781e6cfddc958c5', 'request_id': 'req-020d9272-8989-4455-b587-7f727b2c3407'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:39.573449 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-020d9272-8989-4455-b587-7f727b2c3407 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3511b0d75a7c44278781e6cfddc958c5', 'request_id': 'req-020d9272-8989-4455-b587-7f727b2c3407'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:39.582303 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-020d9272-8989-4455-b587-7f727b2c3407 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3511b0d75a7c44278781e6cfddc958c5', 'request_id': 'req-020d9272-8989-4455-b587-7f727b2c3407'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:39.597478 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-020d9272-8989-4455-b587-7f727b2c3407 admin admin] The token cache is being invalidate because project 3511b0d75a7c44278781e6cfddc958c5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81434) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:41:39.598071 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-020d9272-8989-4455-b587-7f727b2c3407 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:39.598825 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-020d9272-8989-4455-b587-7f727b2c3407 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:39.600484 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1321/2644] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:41:39 2022] DELETE /identity/v3/projects/3511b0d75a7c44278781e6cfddc958c5 => generated 0 bytes in 182 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:41:39.605819 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b1f79f-917f-464d-8ae8-2dfc5d5e015d None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:39.606230 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b1f79f-917f-464d-8ae8-2dfc5d5e015d None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:39.606462 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b1f79f-917f-464d-8ae8-2dfc5d5e015d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:39.629998 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-89b1f79f-917f-464d-8ae8-2dfc5d5e015d None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:41:39.631347 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-89b1f79f-917f-464d-8ae8-2dfc5d5e015d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:39.698417 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1324/2645] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:41:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 21:41:39.706982 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d425c60-df03-403e-a2e8-0f1c4a0cd7d8 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:39.707363 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d425c60-df03-403e-a2e8-0f1c4a0cd7d8 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:39.707707 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d425c60-df03-403e-a2e8-0f1c4a0cd7d8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:39.729425 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-2d425c60-df03-403e-a2e8-0f1c4a0cd7d8 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:41:39.730912 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-2d425c60-df03-403e-a2e8-0f1c4a0cd7d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:39.740092 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1322/2646] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:41:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:41:39.744660 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89b1f79f-917f-464d-8ae8-2dfc5d5e015d None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:39.755880 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89b1f79f-917f-464d-8ae8-2dfc5d5e015d None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:39.760737 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-752ad8d1-98aa-4f05-b429-47ec609fa4d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:39.761375 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-752ad8d1-98aa-4f05-b429-47ec609fa4d2 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:39.761655 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-752ad8d1-98aa-4f05-b429-47ec609fa4d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:39.761974 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-752ad8d1-98aa-4f05-b429-47ec609fa4d2 admin admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:39.762638 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-752ad8d1-98aa-4f05-b429-47ec609fa4d2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:41:39.762945 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-752ad8d1-98aa-4f05-b429-47ec609fa4d2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:39.764567 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-752ad8d1-98aa-4f05-b429-47ec609fa4d2 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:39.781923 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1325/2647] 10.222.0.90 () {58 vars in 1153 bytes} [Mon Aug 29 21:41:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:41:39.783665 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d425c60-df03-403e-a2e8-0f1c4a0cd7d8 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:39.793677 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d425c60-df03-403e-a2e8-0f1c4a0cd7d8 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:39.798995 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2bbc5a0-0cce-4341-95ea-5f68c02a018a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:39.799708 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2bbc5a0-0cce-4341-95ea-5f68c02a018a admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:39.800096 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2bbc5a0-0cce-4341-95ea-5f68c02a018a admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:39.800504 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2bbc5a0-0cce-4341-95ea-5f68c02a018a admin admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:39.801243 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2bbc5a0-0cce-4341-95ea-5f68c02a018a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:39.803309 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2bbc5a0-0cce-4341-95ea-5f68c02a018a admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:39.846156 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1323/2648] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:41:39 2022] POST /identity/v3/projects => generated 371 bytes in 60 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 21:41:39.847819 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-752ad8d1-98aa-4f05-b429-47ec609fa4d2 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:39.863433 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-752ad8d1-98aa-4f05-b429-47ec609fa4d2 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:39.935294 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8739112d-b0c9-4d1a-b95a-3ad7a251f069 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:39.936094 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8739112d-b0c9-4d1a-b95a-3ad7a251f069 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:39.936423 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8739112d-b0c9-4d1a-b95a-3ad7a251f069 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:39.936835 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8739112d-b0c9-4d1a-b95a-3ad7a251f069 admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:39.937649 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8739112d-b0c9-4d1a-b95a-3ad7a251f069 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:39.940547 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8739112d-b0c9-4d1a-b95a-3ad7a251f069 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:39.951193 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:41:39.975951 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1326/2649] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:41:39 2022] POST /identity/v3/users => generated 286 bytes in 128 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:41:39.976488 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2bbc5a0-0cce-4341-95ea-5f68c02a018a admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:39.991286 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2bbc5a0-0cce-4341-95ea-5f68c02a018a admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:39.999646 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bdf78a0-0355-4274-8118-d20debc23eec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:40.000481 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bdf78a0-0355-4274-8118-d20debc23eec admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:40.001177 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bdf78a0-0355-4274-8118-d20debc23eec admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:40.001463 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bdf78a0-0355-4274-8118-d20debc23eec admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:40.002503 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bdf78a0-0355-4274-8118-d20debc23eec admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:40.004150 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bdf78a0-0355-4274-8118-d20debc23eec admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:40.017894 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1324/2650] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:41:39 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:41:40.019907 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8739112d-b0c9-4d1a-b95a-3ad7a251f069 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:40.030294 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8739112d-b0c9-4d1a-b95a-3ad7a251f069 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:40.035200 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba7be920-e051-41d5-b001-a0a46aa2d3cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:40.035819 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba7be920-e051-41d5-b001-a0a46aa2d3cd admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:40.036104 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba7be920-e051-41d5-b001-a0a46aa2d3cd admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:40.036377 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba7be920-e051-41d5-b001-a0a46aa2d3cd admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:40.037040 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba7be920-e051-41d5-b001-a0a46aa2d3cd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:40.038706 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba7be920-e051-41d5-b001-a0a46aa2d3cd admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:40.053829 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1327/2651] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:41:40 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 29 21:41:40.055901 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bdf78a0-0355-4274-8118-d20debc23eec admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:40.067278 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bdf78a0-0355-4274-8118-d20debc23eec admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:40.071859 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-638d4ad6-eb8e-4c38-a929-8379c147bf47 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:40.072524 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-638d4ad6-eb8e-4c38-a929-8379c147bf47 admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:40.072805 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-638d4ad6-eb8e-4c38-a929-8379c147bf47 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:40.073080 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-638d4ad6-eb8e-4c38-a929-8379c147bf47 admin admin] PATH_INFO: `/v3/projects/600f0b4954144efb9725460eb1ebaa1f/users/6f91ae9fecb7421297dd09a58514532c/roles/02e242d5520740d4b77a5f4008c7eb57` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:40.090722 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-638d4ad6-eb8e-4c38-a929-8379c147bf47 admin admin] RBAC: Authorizing `identity:create_grant(project_id=600f0b4954144efb9725460eb1ebaa1f, user_id=6f91ae9fecb7421297dd09a58514532c, role_id=02e242d5520740d4b77a5f4008c7eb57)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:40.093927 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-638d4ad6-eb8e-4c38-a929-8379c147bf47 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:40.113862 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1325/2652] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:41:40 2022] PUT /identity/v3/projects/600f0b4954144efb9725460eb1ebaa1f/users/6f91ae9fecb7421297dd09a58514532c/roles/02e242d5520740d4b77a5f4008c7eb57 => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:41:40.116793 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77016fbb-66ca-4d79-a54e-892fa95dfb95 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:40.117273 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77016fbb-66ca-4d79-a54e-892fa95dfb95 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:40.117492 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77016fbb-66ca-4d79-a54e-892fa95dfb95 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:40.138953 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-77016fbb-66ca-4d79-a54e-892fa95dfb95 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:41:40.140163 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-77016fbb-66ca-4d79-a54e-892fa95dfb95 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:40.209754 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1328/2653] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:41:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:41:40.219341 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-638d4ad6-eb8e-4c38-a929-8379c147bf47 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:40.221454 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-638d4ad6-eb8e-4c38-a929-8379c147bf47 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:40.242003 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=88,keystone:INSERT=4,keystone:DELETE=8 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:41:40.284070 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-638d4ad6-eb8e-4c38-a929-8379c147bf47 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:40.287891 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46dc6ea6-14a5-49a7-9032-ed9164da4208 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:40.288511 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46dc6ea6-14a5-49a7-9032-ed9164da4208 service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:40.288793 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46dc6ea6-14a5-49a7-9032-ed9164da4208 service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:40.289068 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46dc6ea6-14a5-49a7-9032-ed9164da4208 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:40.298752 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46dc6ea6-14a5-49a7-9032-ed9164da4208 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:40.300213 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46dc6ea6-14a5-49a7-9032-ed9164da4208 service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:40.310121 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1326/2654] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:41:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:41:40.959942 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd2f38df-b452-4476-a3c0-8bd295c12f52 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:40.959942 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd2f38df-b452-4476-a3c0-8bd295c12f52 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:40.959942 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd2f38df-b452-4476-a3c0-8bd295c12f52 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:40.978284 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9499d07-6c8b-4e07-a3cf-c5a0ce034177 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:40.978803 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9499d07-6c8b-4e07-a3cf-c5a0ce034177 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:40.978911 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9499d07-6c8b-4e07-a3cf-c5a0ce034177 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:40.997282 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-cd2f38df-b452-4476-a3c0-8bd295c12f52 None None] MFA Rules not processed for user `6f91ae9fecb7421297dd09a58514532c`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:41:40.998703 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-cd2f38df-b452-4476-a3c0-8bd295c12f52 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:41.000098 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-a9499d07-6c8b-4e07-a3cf-c5a0ce034177 None None] MFA Rules not processed for user `7952b085a8b94dcbbeb6a2817ad602e9`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:41:41.001360 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-a9499d07-6c8b-4e07-a3cf-c5a0ce034177 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:41.064826 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1327/2655] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:41:40 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 21:41:41.084479 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1329/2656] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:41:40 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 21:41:41.085397 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9499d07-6c8b-4e07-a3cf-c5a0ce034177 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:41.097009 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9499d07-6c8b-4e07-a3cf-c5a0ce034177 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:41.103513 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71e54d21-2eda-4916-9abb-32d0c46331b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:41.104673 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71e54d21-2eda-4916-9abb-32d0c46331b7 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:41.104673 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71e54d21-2eda-4916-9abb-32d0c46331b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:41.104920 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71e54d21-2eda-4916-9abb-32d0c46331b7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:41.105643 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71e54d21-2eda-4916-9abb-32d0c46331b7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:41.107757 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71e54d21-2eda-4916-9abb-32d0c46331b7 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:41.145839 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1328/2657] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:41:41 2022] POST /identity/v3/projects => generated 373 bytes in 58 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 21:41:41.151916 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd2f38df-b452-4476-a3c0-8bd295c12f52 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:41.163397 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd2f38df-b452-4476-a3c0-8bd295c12f52 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:41.222002 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bfa9b17-1ef7-4756-a5a7-c51fbfbf416f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:41.222694 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bfa9b17-1ef7-4756-a5a7-c51fbfbf416f admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:41.222890 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bfa9b17-1ef7-4756-a5a7-c51fbfbf416f admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:41.223165 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bfa9b17-1ef7-4756-a5a7-c51fbfbf416f admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:41.223822 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bfa9b17-1ef7-4756-a5a7-c51fbfbf416f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:41.225748 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bfa9b17-1ef7-4756-a5a7-c51fbfbf416f admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:41.244556 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:INSERT=5,keystone:SELECT=95 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:41:41.254242 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1330/2658] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:41:41 2022] POST /identity/v3/users => generated 293 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 21:41:41.255402 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71e54d21-2eda-4916-9abb-32d0c46331b7 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:41.266170 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71e54d21-2eda-4916-9abb-32d0c46331b7 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:41.271761 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e2b2bdf-42c0-4e71-90a1-9e245ea50c1c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:41.272386 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e2b2bdf-42c0-4e71-90a1-9e245ea50c1c admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:41.272670 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e2b2bdf-42c0-4e71-90a1-9e245ea50c1c admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:41.272946 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e2b2bdf-42c0-4e71-90a1-9e245ea50c1c admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:41.273595 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e2b2bdf-42c0-4e71-90a1-9e245ea50c1c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:41.275262 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e2b2bdf-42c0-4e71-90a1-9e245ea50c1c admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:41.286048 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1329/2659] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:41:41 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:41:41.290014 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bfa9b17-1ef7-4756-a5a7-c51fbfbf416f admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:41.300608 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bfa9b17-1ef7-4756-a5a7-c51fbfbf416f admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:41.305354 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c07ca889-18f6-48dd-baf5-db99a1322dc3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:41.306051 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c07ca889-18f6-48dd-baf5-db99a1322dc3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:41.306337 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c07ca889-18f6-48dd-baf5-db99a1322dc3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:41.306635 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c07ca889-18f6-48dd-baf5-db99a1322dc3 admin admin] PATH_INFO: `/v3/projects/e33fac7e79504d0499b441205e197202/users/309b52e95ab74e89a201bbe12010cd44/roles/a6e562cdc96c4be691d638990f8403fd` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:41.323651 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c07ca889-18f6-48dd-baf5-db99a1322dc3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e33fac7e79504d0499b441205e197202, user_id=309b52e95ab74e89a201bbe12010cd44, role_id=a6e562cdc96c4be691d638990f8403fd)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:41.326729 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c07ca889-18f6-48dd-baf5-db99a1322dc3 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:41.341222 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1331/2660] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:41:41 2022] PUT /identity/v3/projects/e33fac7e79504d0499b441205e197202/users/309b52e95ab74e89a201bbe12010cd44/roles/a6e562cdc96c4be691d638990f8403fd => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:41:41.345332 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e2b2bdf-42c0-4e71-90a1-9e245ea50c1c admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:41.356673 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e2b2bdf-42c0-4e71-90a1-9e245ea50c1c admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:41.425138 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2acaaf85-257a-4159-9d2d-40e0f3063bcb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:41.425766 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2acaaf85-257a-4159-9d2d-40e0f3063bcb admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:41.426080 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2acaaf85-257a-4159-9d2d-40e0f3063bcb admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:41.426434 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2acaaf85-257a-4159-9d2d-40e0f3063bcb admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:41.427101 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2acaaf85-257a-4159-9d2d-40e0f3063bcb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:41.428769 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2acaaf85-257a-4159-9d2d-40e0f3063bcb admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:41.442626 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1330/2661] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:41:41 2022] GET /identity/v3/roles => generated 1420 bytes in 94 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:41:41.443807 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c07ca889-18f6-48dd-baf5-db99a1322dc3 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:41.454638 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c07ca889-18f6-48dd-baf5-db99a1322dc3 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:41.459454 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0fbbf53-135f-4aad-ad03-e28b4eee9753 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:41.460120 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0fbbf53-135f-4aad-ad03-e28b4eee9753 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:41.460407 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0fbbf53-135f-4aad-ad03-e28b4eee9753 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:41.460687 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0fbbf53-135f-4aad-ad03-e28b4eee9753 admin admin] PATH_INFO: `/v3/projects/e33fac7e79504d0499b441205e197202/users/309b52e95ab74e89a201bbe12010cd44/roles/a6e562cdc96c4be691d638990f8403fd` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:41.462674 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0fbbf53-135f-4aad-ad03-e28b4eee9753 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e33fac7e79504d0499b441205e197202, user_id=309b52e95ab74e89a201bbe12010cd44, role_id=a6e562cdc96c4be691d638990f8403fd)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:41.465382 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0fbbf53-135f-4aad-ad03-e28b4eee9753 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:41.478044 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1332/2662] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:41:41 2022] PUT /identity/v3/projects/e33fac7e79504d0499b441205e197202/users/309b52e95ab74e89a201bbe12010cd44/roles/a6e562cdc96c4be691d638990f8403fd => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:41:42.183116 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6485bf6b-fffe-420b-b411-3b398acce3ac None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:42.183728 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6485bf6b-fffe-420b-b411-3b398acce3ac None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:42.183728 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6485bf6b-fffe-420b-b411-3b398acce3ac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:42.216168 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-6485bf6b-fffe-420b-b411-3b398acce3ac None None] MFA Rules not processed for user `309b52e95ab74e89a201bbe12010cd44`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:41:42.217406 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-6485bf6b-fffe-420b-b411-3b398acce3ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:42.233028 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:41:42.306224 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1331/2663] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:41:42 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:41:42.317991 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0fbbf53-135f-4aad-ad03-e28b4eee9753 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:42.320046 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-f0fbbf53-135f-4aad-ad03-e28b4eee9753 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:42.389768 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0fbbf53-135f-4aad-ad03-e28b4eee9753 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:42.394350 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88adda21-5b2c-45c1-ae3e-4bd0ad748048 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dc226f7ddcb49ecacb51ce57fdfc11b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:42.394899 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88adda21-5b2c-45c1-ae3e-4bd0ad748048 service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:42.395181 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88adda21-5b2c-45c1-ae3e-4bd0ad748048 service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:42.395661 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88adda21-5b2c-45c1-ae3e-4bd0ad748048 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:42.456734 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88adda21-5b2c-45c1-ae3e-4bd0ad748048 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:42.458222 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88adda21-5b2c-45c1-ae3e-4bd0ad748048 service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:42.467570 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1333/2664] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:41:42 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 150 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:41:43.846440 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6485bf6b-fffe-420b-b411-3b398acce3ac None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:43.848908 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-6485bf6b-fffe-420b-b411-3b398acce3ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:43.861239 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6485bf6b-fffe-420b-b411-3b398acce3ac None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:43.866233 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd7c0c33-5547-4694-9038-e1667045fcf4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dc226f7ddcb49ecacb51ce57fdfc11b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:43.866855 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd7c0c33-5547-4694-9038-e1667045fcf4 service cinder] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:43.867137 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd7c0c33-5547-4694-9038-e1667045fcf4 service cinder] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:43.867407 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd7c0c33-5547-4694-9038-e1667045fcf4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:43.878254 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd7c0c33-5547-4694-9038-e1667045fcf4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:43.879377 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd7c0c33-5547-4694-9038-e1667045fcf4 service cinder] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:43.901483 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1332/2665] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:41:43 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 56 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:41:52.463408 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=63,keystone:INSERT=2 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:41:53.885732 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=28 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:41:57.621113 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88adda21-5b2c-45c1-ae3e-4bd0ad748048 service cinder] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:57.690611 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88adda21-5b2c-45c1-ae3e-4bd0ad748048 service cinder] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:57.695387 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d1fae53-d4cc-4251-bf69-8dc050685e97 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:57.696033 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d1fae53-d4cc-4251-bf69-8dc050685e97 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:57.696312 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d1fae53-d4cc-4251-bf69-8dc050685e97 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:57.696586 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d1fae53-d4cc-4251-bf69-8dc050685e97 admin admin] PATH_INFO: `/v3/users/6f91ae9fecb7421297dd09a58514532c` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:57.697748 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d1fae53-d4cc-4251-bf69-8dc050685e97 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6f91ae9fecb7421297dd09a58514532c)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:57.699724 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d1fae53-d4cc-4251-bf69-8dc050685e97 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:57.754599 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-0d1fae53-d4cc-4251-bf69-8dc050685e97 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6f91ae9fecb7421297dd09a58514532c'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:57.768101 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-0d1fae53-d4cc-4251-bf69-8dc050685e97 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6f91ae9fecb7421297dd09a58514532c'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:57.769118 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-0d1fae53-d4cc-4251-bf69-8dc050685e97 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6f91ae9fecb7421297dd09a58514532c'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:57.769752 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-0d1fae53-d4cc-4251-bf69-8dc050685e97 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6f91ae9fecb7421297dd09a58514532c'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:57.777264 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-0d1fae53-d4cc-4251-bf69-8dc050685e97 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6f91ae9fecb7421297dd09a58514532c'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:57.790477 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1334/2666] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:41:57 2022] DELETE /identity/v3/users/6f91ae9fecb7421297dd09a58514532c => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 21:41:57.993076 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd7c0c33-5547-4694-9038-e1667045fcf4 service cinder] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:58.004256 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd7c0c33-5547-4694-9038-e1667045fcf4 service cinder] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:58.082382 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d649a84-ad7a-4f63-a2cc-b47723d94610 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:58.083514 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d649a84-ad7a-4f63-a2cc-b47723d94610 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:58.083514 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d649a84-ad7a-4f63-a2cc-b47723d94610 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:58.083724 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d649a84-ad7a-4f63-a2cc-b47723d94610 admin admin] PATH_INFO: `/v3/projects/600f0b4954144efb9725460eb1ebaa1f` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:58.084809 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d649a84-ad7a-4f63-a2cc-b47723d94610 admin admin] RBAC: Authorizing `identity:delete_project(project_id=600f0b4954144efb9725460eb1ebaa1f)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:58.086724 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d649a84-ad7a-4f63-a2cc-b47723d94610 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:58.139419 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-3d649a84-ad7a-4f63-a2cc-b47723d94610 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '600f0b4954144efb9725460eb1ebaa1f', 'request_id': 'req-3d649a84-ad7a-4f63-a2cc-b47723d94610'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:58.144473 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-3d649a84-ad7a-4f63-a2cc-b47723d94610 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '600f0b4954144efb9725460eb1ebaa1f', 'request_id': 'req-3d649a84-ad7a-4f63-a2cc-b47723d94610'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:58.150882 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-3d649a84-ad7a-4f63-a2cc-b47723d94610 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '600f0b4954144efb9725460eb1ebaa1f', 'request_id': 'req-3d649a84-ad7a-4f63-a2cc-b47723d94610'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:58.169950 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-3d649a84-ad7a-4f63-a2cc-b47723d94610 admin admin] The token cache is being invalidate because project 600f0b4954144efb9725460eb1ebaa1f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81434) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:41:58.170306 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-3d649a84-ad7a-4f63-a2cc-b47723d94610 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:58.170899 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-3d649a84-ad7a-4f63-a2cc-b47723d94610 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:58.173018 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1333/2667] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:41:57 2022] DELETE /identity/v3/projects/600f0b4954144efb9725460eb1ebaa1f => generated 0 bytes in 180 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:41:58.177440 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d1fae53-d4cc-4251-bf69-8dc050685e97 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:58.178891 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-0d1fae53-d4cc-4251-bf69-8dc050685e97 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:58.192974 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d1fae53-d4cc-4251-bf69-8dc050685e97 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:58.265990 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db763ad5-7dfe-489e-9332-f2277e450c93 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:58.267103 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db763ad5-7dfe-489e-9332-f2277e450c93 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:58.267387 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db763ad5-7dfe-489e-9332-f2277e450c93 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:58.267659 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db763ad5-7dfe-489e-9332-f2277e450c93 admin admin] PATH_INFO: `/v3/users/309b52e95ab74e89a201bbe12010cd44` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:58.268859 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db763ad5-7dfe-489e-9332-f2277e450c93 admin admin] RBAC: Authorizing `identity:delete_user(user_id=309b52e95ab74e89a201bbe12010cd44)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:58.271076 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db763ad5-7dfe-489e-9332-f2277e450c93 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:58.295107 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:41:58.323032 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-db763ad5-7dfe-489e-9332-f2277e450c93 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '309b52e95ab74e89a201bbe12010cd44'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:58.339986 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-db763ad5-7dfe-489e-9332-f2277e450c93 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '309b52e95ab74e89a201bbe12010cd44'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:58.340744 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-db763ad5-7dfe-489e-9332-f2277e450c93 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '309b52e95ab74e89a201bbe12010cd44'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:58.341373 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-db763ad5-7dfe-489e-9332-f2277e450c93 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '309b52e95ab74e89a201bbe12010cd44'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:58.351178 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-db763ad5-7dfe-489e-9332-f2277e450c93 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '309b52e95ab74e89a201bbe12010cd44'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:58.372340 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1335/2668] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:41:58 2022] DELETE /identity/v3/users/309b52e95ab74e89a201bbe12010cd44 => generated 0 bytes in 195 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:41:58.562179 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d649a84-ad7a-4f63-a2cc-b47723d94610 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:58.563681 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-3d649a84-ad7a-4f63-a2cc-b47723d94610 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:58.577372 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d649a84-ad7a-4f63-a2cc-b47723d94610 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:58.642235 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb16bda3-bc71-45a9-ba40-1f38aa5daeab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:58.643116 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb16bda3-bc71-45a9-ba40-1f38aa5daeab admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:58.643425 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb16bda3-bc71-45a9-ba40-1f38aa5daeab admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:58.643702 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb16bda3-bc71-45a9-ba40-1f38aa5daeab admin admin] PATH_INFO: `/v3/projects/e33fac7e79504d0499b441205e197202` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:58.644918 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb16bda3-bc71-45a9-ba40-1f38aa5daeab admin admin] RBAC: Authorizing `identity:delete_project(project_id=e33fac7e79504d0499b441205e197202)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:58.646903 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb16bda3-bc71-45a9-ba40-1f38aa5daeab admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:58.686280 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:41:58.700170 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-bb16bda3-bc71-45a9-ba40-1f38aa5daeab admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e33fac7e79504d0499b441205e197202', 'request_id': 'req-bb16bda3-bc71-45a9-ba40-1f38aa5daeab'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:58.706600 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-bb16bda3-bc71-45a9-ba40-1f38aa5daeab admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e33fac7e79504d0499b441205e197202', 'request_id': 'req-bb16bda3-bc71-45a9-ba40-1f38aa5daeab'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:58.712819 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-bb16bda3-bc71-45a9-ba40-1f38aa5daeab admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e33fac7e79504d0499b441205e197202', 'request_id': 'req-bb16bda3-bc71-45a9-ba40-1f38aa5daeab'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:58.727747 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-bb16bda3-bc71-45a9-ba40-1f38aa5daeab admin admin] The token cache is being invalidate because project e33fac7e79504d0499b441205e197202 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81434) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:41:58.728040 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-bb16bda3-bc71-45a9-ba40-1f38aa5daeab admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:58.729567 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-bb16bda3-bc71-45a9-ba40-1f38aa5daeab admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:41:58.731150 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1334/2669] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:41:58 2022] DELETE /identity/v3/projects/e33fac7e79504d0499b441205e197202 => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:41:58.737703 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0104945-a43c-4c17-a01d-216e2122b3eb None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:58.737703 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0104945-a43c-4c17-a01d-216e2122b3eb None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:58.737703 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0104945-a43c-4c17-a01d-216e2122b3eb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:58.759000 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-f0104945-a43c-4c17-a01d-216e2122b3eb None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:41:58.760212 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-f0104945-a43c-4c17-a01d-216e2122b3eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:58.833415 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1336/2670] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:41:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:41:58.841851 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d5857ab-3f80-4715-9710-0cce715b3c9d None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:58.842135 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d5857ab-3f80-4715-9710-0cce715b3c9d None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:58.842428 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d5857ab-3f80-4715-9710-0cce715b3c9d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:58.864341 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-5d5857ab-3f80-4715-9710-0cce715b3c9d None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:41:58.865581 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-5d5857ab-3f80-4715-9710-0cce715b3c9d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:58.874549 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1335/2671] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:41:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:41:58.879611 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0104945-a43c-4c17-a01d-216e2122b3eb None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:58.890399 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0104945-a43c-4c17-a01d-216e2122b3eb None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:58.897597 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb9bd8af-bbac-473d-a2fa-b8044422bedc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:58.898281 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb9bd8af-bbac-473d-a2fa-b8044422bedc admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:58.898564 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb9bd8af-bbac-473d-a2fa-b8044422bedc admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:58.898841 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb9bd8af-bbac-473d-a2fa-b8044422bedc admin admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:58.899508 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb9bd8af-bbac-473d-a2fa-b8044422bedc admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:41:58.899814 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb9bd8af-bbac-473d-a2fa-b8044422bedc admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:58.901716 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb9bd8af-bbac-473d-a2fa-b8044422bedc admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:58.917669 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1337/2672] 10.222.0.90 () {58 vars in 1153 bytes} [Mon Aug 29 21:41:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:41:58.922196 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d5857ab-3f80-4715-9710-0cce715b3c9d None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:58.932658 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d5857ab-3f80-4715-9710-0cce715b3c9d None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:58.939018 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe0b1da8-1e49-4e71-9470-3dc7d19aca06 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:58.939553 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe0b1da8-1e49-4e71-9470-3dc7d19aca06 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:58.939839 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe0b1da8-1e49-4e71-9470-3dc7d19aca06 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:58.940112 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe0b1da8-1e49-4e71-9470-3dc7d19aca06 admin admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:58.940756 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe0b1da8-1e49-4e71-9470-3dc7d19aca06 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:58.943760 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe0b1da8-1e49-4e71-9470-3dc7d19aca06 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:58.978779 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1336/2673] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:41:58 2022] POST /identity/v3/projects => generated 385 bytes in 57 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:41:58.983615 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb9bd8af-bbac-473d-a2fa-b8044422bedc admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:58.995405 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb9bd8af-bbac-473d-a2fa-b8044422bedc admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:59.070134 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2334353e-0023-4cf5-8d94-bf9d9b6915a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:59.071100 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2334353e-0023-4cf5-8d94-bf9d9b6915a3 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:59.071100 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2334353e-0023-4cf5-8d94-bf9d9b6915a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:59.071305 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2334353e-0023-4cf5-8d94-bf9d9b6915a3 admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:59.072086 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2334353e-0023-4cf5-8d94-bf9d9b6915a3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:59.074417 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2334353e-0023-4cf5-8d94-bf9d9b6915a3 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:59.105717 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1338/2674] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:41:58 2022] POST /identity/v3/users => generated 293 bytes in 120 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 21:41:59.107477 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe0b1da8-1e49-4e71-9470-3dc7d19aca06 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:59.119434 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe0b1da8-1e49-4e71-9470-3dc7d19aca06 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:59.124995 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f93c5f07-75d2-4757-82bc-6dc32a519a7d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:59.126256 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f93c5f07-75d2-4757-82bc-6dc32a519a7d admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:59.126545 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f93c5f07-75d2-4757-82bc-6dc32a519a7d admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:59.126817 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f93c5f07-75d2-4757-82bc-6dc32a519a7d admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:59.127494 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f93c5f07-75d2-4757-82bc-6dc32a519a7d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:59.129270 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f93c5f07-75d2-4757-82bc-6dc32a519a7d admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:59.142976 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1337/2675] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:41:59 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:41:59.150076 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2334353e-0023-4cf5-8d94-bf9d9b6915a3 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:59.161051 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2334353e-0023-4cf5-8d94-bf9d9b6915a3 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:59.166078 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-671a373a-480d-47a5-8f34-2017ca21432e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:59.166725 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-671a373a-480d-47a5-8f34-2017ca21432e admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:59.166980 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-671a373a-480d-47a5-8f34-2017ca21432e admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:59.167260 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-671a373a-480d-47a5-8f34-2017ca21432e admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:59.167913 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-671a373a-480d-47a5-8f34-2017ca21432e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:59.169568 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-671a373a-480d-47a5-8f34-2017ca21432e admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:59.182617 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1339/2676] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:41:59 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:41:59.187251 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f93c5f07-75d2-4757-82bc-6dc32a519a7d admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:59.201246 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f93c5f07-75d2-4757-82bc-6dc32a519a7d admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:59.207975 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da506206-7dcc-4006-a2da-4886e420a913 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:59.207975 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da506206-7dcc-4006-a2da-4886e420a913 admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:59.208511 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da506206-7dcc-4006-a2da-4886e420a913 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:59.208511 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da506206-7dcc-4006-a2da-4886e420a913 admin admin] PATH_INFO: `/v3/projects/38ffe55e1bf34a21af1add5945c01761/users/cd8e9c9275144a339a18808375ba8e33/roles/02e242d5520740d4b77a5f4008c7eb57` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:59.224514 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da506206-7dcc-4006-a2da-4886e420a913 admin admin] RBAC: Authorizing `identity:create_grant(project_id=38ffe55e1bf34a21af1add5945c01761, user_id=cd8e9c9275144a339a18808375ba8e33, role_id=02e242d5520740d4b77a5f4008c7eb57)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:59.227182 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da506206-7dcc-4006-a2da-4886e420a913 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:59.242188 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1338/2677] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:41:59 2022] PUT /identity/v3/projects/38ffe55e1bf34a21af1add5945c01761/users/cd8e9c9275144a339a18808375ba8e33/roles/02e242d5520740d4b77a5f4008c7eb57 => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:41:59.255199 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df34777f-27a5-4e8a-ab63-dcd7895ea3a5 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:59.255488 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df34777f-27a5-4e8a-ab63-dcd7895ea3a5 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:59.255778 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df34777f-27a5-4e8a-ab63-dcd7895ea3a5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:59.264073 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:DELETE=7,keystone:SELECT=89,keystone:INSERT=4 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:41:59.290279 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-df34777f-27a5-4e8a-ab63-dcd7895ea3a5 None None] MFA Rules not processed for user `cd8e9c9275144a339a18808375ba8e33`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:41:59.291541 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-df34777f-27a5-4e8a-ab63-dcd7895ea3a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:59.367337 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1340/2678] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:41:59 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:41:59.374389 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da506206-7dcc-4006-a2da-4886e420a913 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:59.447468 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da506206-7dcc-4006-a2da-4886e420a913 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:59.452683 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4d01b43-17c6-4bd6-b640-446cb0b9958e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:59.453263 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d01b43-17c6-4bd6-b640-446cb0b9958e admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:59.453642 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d01b43-17c6-4bd6-b640-446cb0b9958e admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:59.453960 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d01b43-17c6-4bd6-b640-446cb0b9958e admin admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:59.454609 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4d01b43-17c6-4bd6-b640-446cb0b9958e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:59.457447 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4d01b43-17c6-4bd6-b640-446cb0b9958e admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:59.489639 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1339/2679] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:41:59 2022] POST /identity/v3/projects => generated 385 bytes in 116 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:41:59.493965 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df34777f-27a5-4e8a-ab63-dcd7895ea3a5 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:59.505580 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df34777f-27a5-4e8a-ab63-dcd7895ea3a5 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:59.569350 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52783e86-19a9-4569-9068-cdbd455cf80e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:59.570059 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52783e86-19a9-4569-9068-cdbd455cf80e admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:59.570341 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52783e86-19a9-4569-9068-cdbd455cf80e admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:59.570610 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52783e86-19a9-4569-9068-cdbd455cf80e admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:59.571259 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52783e86-19a9-4569-9068-cdbd455cf80e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:59.573180 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52783e86-19a9-4569-9068-cdbd455cf80e admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:59.597083 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1341/2680] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:41:59 2022] POST /identity/v3/users => generated 299 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:41:59.601736 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4d01b43-17c6-4bd6-b640-446cb0b9958e admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:59.620399 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4d01b43-17c6-4bd6-b640-446cb0b9958e admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:59.625097 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa43b01b-6db8-4325-b962-f191693c7fb8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:59.625809 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa43b01b-6db8-4325-b962-f191693c7fb8 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:59.626092 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa43b01b-6db8-4325-b962-f191693c7fb8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:59.626362 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa43b01b-6db8-4325-b962-f191693c7fb8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:59.626999 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa43b01b-6db8-4325-b962-f191693c7fb8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:59.628574 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa43b01b-6db8-4325-b962-f191693c7fb8 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:59.639049 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1340/2681] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:41:59 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:41:59.643384 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52783e86-19a9-4569-9068-cdbd455cf80e admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:59.653631 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52783e86-19a9-4569-9068-cdbd455cf80e admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:59.659071 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e9304bc-2d95-4aaa-a266-05899d071030 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:59.659721 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e9304bc-2d95-4aaa-a266-05899d071030 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:59.660002 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e9304bc-2d95-4aaa-a266-05899d071030 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:59.660276 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e9304bc-2d95-4aaa-a266-05899d071030 admin admin] PATH_INFO: `/v3/projects/2e420d2c67c94d97b369b2c8f8e26530/users/beb3c60c4f1b414dbbbd99d5e76050b5/roles/a6e562cdc96c4be691d638990f8403fd` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:59.676189 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e9304bc-2d95-4aaa-a266-05899d071030 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2e420d2c67c94d97b369b2c8f8e26530, user_id=beb3c60c4f1b414dbbbd99d5e76050b5, role_id=a6e562cdc96c4be691d638990f8403fd)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:59.679054 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e9304bc-2d95-4aaa-a266-05899d071030 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:59.695669 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1342/2682] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:41:59 2022] PUT /identity/v3/projects/2e420d2c67c94d97b369b2c8f8e26530/users/beb3c60c4f1b414dbbbd99d5e76050b5/roles/a6e562cdc96c4be691d638990f8403fd => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:41:59.699772 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa43b01b-6db8-4325-b962-f191693c7fb8 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:59.710434 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa43b01b-6db8-4325-b962-f191693c7fb8 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:59.740603 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=91,keystone:DELETE=5,keystone:INSERT=4 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:41:59.769773 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8631b4f9-1284-4988-a703-756c821c8747 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:59.770442 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8631b4f9-1284-4988-a703-756c821c8747 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:59.770729 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8631b4f9-1284-4988-a703-756c821c8747 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:59.770993 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8631b4f9-1284-4988-a703-756c821c8747 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:59.771635 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8631b4f9-1284-4988-a703-756c821c8747 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:59.773242 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8631b4f9-1284-4988-a703-756c821c8747 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:59.783850 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1341/2683] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:41:59 2022] GET /identity/v3/roles => generated 1420 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 29 21:41:59.788162 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e9304bc-2d95-4aaa-a266-05899d071030 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:59.799791 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e9304bc-2d95-4aaa-a266-05899d071030 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:41:59.804472 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-747d27db-4bc4-479a-a124-6c3e24a62467 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:41:59.805158 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-747d27db-4bc4-479a-a124-6c3e24a62467 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:59.805437 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-747d27db-4bc4-479a-a124-6c3e24a62467 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:59.805709 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-747d27db-4bc4-479a-a124-6c3e24a62467 admin admin] PATH_INFO: `/v3/projects/2e420d2c67c94d97b369b2c8f8e26530/users/beb3c60c4f1b414dbbbd99d5e76050b5/roles/a6e562cdc96c4be691d638990f8403fd` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:59.807651 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-747d27db-4bc4-479a-a124-6c3e24a62467 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2e420d2c67c94d97b369b2c8f8e26530, user_id=beb3c60c4f1b414dbbbd99d5e76050b5, role_id=a6e562cdc96c4be691d638990f8403fd)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:41:59.810333 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-747d27db-4bc4-479a-a124-6c3e24a62467 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:41:59.822065 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1343/2684] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:41:59 2022] PUT /identity/v3/projects/2e420d2c67c94d97b369b2c8f8e26530/users/beb3c60c4f1b414dbbbd99d5e76050b5/roles/a6e562cdc96c4be691d638990f8403fd => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:41:59.834138 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6062821f-306c-4070-8750-055e278d73b1 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:59.834435 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6062821f-306c-4070-8750-055e278d73b1 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:59.835081 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6062821f-306c-4070-8750-055e278d73b1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:59.869516 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-6062821f-306c-4070-8750-055e278d73b1 None None] MFA Rules not processed for user `beb3c60c4f1b414dbbbd99d5e76050b5`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:41:59.885818 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-6062821f-306c-4070-8750-055e278d73b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:59.903976 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef0dfb35-7b59-48bd-8497-d60d3fca63fc None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:41:59.904268 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef0dfb35-7b59-48bd-8497-d60d3fca63fc None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:41:59.904549 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef0dfb35-7b59-48bd-8497-d60d3fca63fc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:41:59.937072 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-ef0dfb35-7b59-48bd-8497-d60d3fca63fc None None] MFA Rules not processed for user `0c0b0da87b124de1be00faec4bbdb404`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:41:59.938482 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-ef0dfb35-7b59-48bd-8497-d60d3fca63fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:59.952787 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1342/2685] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:41:59 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:41:59.964590 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6062821f-306c-4070-8750-055e278d73b1 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:41:59.965953 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-6062821f-306c-4070-8750-055e278d73b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:41:59.977278 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:42:00.009667 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1344/2686] 10.222.0.90 () {58 vars in 1009 bytes} [Mon Aug 29 21:41:59 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 21:42:00.033959 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef0dfb35-7b59-48bd-8497-d60d3fca63fc None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:00.035470 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-ef0dfb35-7b59-48bd-8497-d60d3fca63fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:00.054030 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6062821f-306c-4070-8750-055e278d73b1 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:00.057777 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11f0e68d-62f4-4c41-89c3-a31e620b1aa6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dc226f7ddcb49ecacb51ce57fdfc11b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:00.058421 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11f0e68d-62f4-4c41-89c3-a31e620b1aa6 service cinder] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:00.058703 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11f0e68d-62f4-4c41-89c3-a31e620b1aa6 service cinder] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:00.059007 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11f0e68d-62f4-4c41-89c3-a31e620b1aa6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:00.121274 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:42:00.136261 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11f0e68d-62f4-4c41-89c3-a31e620b1aa6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:00.137800 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11f0e68d-62f4-4c41-89c3-a31e620b1aa6 service cinder] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:00.148264 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1343/2687] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:41:59 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 184 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 21:42:00.236683 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef0dfb35-7b59-48bd-8497-d60d3fca63fc None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:00.243270 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44632fe5-30dd-42c6-89d7-abc6c721b01d service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '139bbbde6d884b42b5440727926e2c58', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:00.243892 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44632fe5-30dd-42c6-89d7-abc6c721b01d service swift] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:00.244171 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44632fe5-30dd-42c6-89d7-abc6c721b01d service swift] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:00.244470 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44632fe5-30dd-42c6-89d7-abc6c721b01d service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:00.256863 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44632fe5-30dd-42c6-89d7-abc6c721b01d service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:00.259600 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44632fe5-30dd-42c6-89d7-abc6c721b01d service swift] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:00.268346 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1345/2688] 10.222.0.90 () {60 vars in 1384 bytes} [Mon Aug 29 21:42:00 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 235 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 29 21:42:03.594863 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11f0e68d-62f4-4c41-89c3-a31e620b1aa6 service cinder] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:03.596460 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-11f0e68d-62f4-4c41-89c3-a31e620b1aa6 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:03.732948 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11f0e68d-62f4-4c41-89c3-a31e620b1aa6 service cinder] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:03.739621 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fe23d71-0456-45ca-8149-76eae6da85ce service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:03.739621 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fe23d71-0456-45ca-8149-76eae6da85ce service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:03.739949 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fe23d71-0456-45ca-8149-76eae6da85ce service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:03.740198 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fe23d71-0456-45ca-8149-76eae6da85ce service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:03.742856 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-2fe23d71-0456-45ca-8149-76eae6da85ce service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:03.831070 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fe23d71-0456-45ca-8149-76eae6da85ce service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:03.833268 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fe23d71-0456-45ca-8149-76eae6da85ce service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:03.857092 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1344/2689] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:42:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 263 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:42:04.837368 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44632fe5-30dd-42c6-89d7-abc6c721b01d service swift] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:04.851897 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44632fe5-30dd-42c6-89d7-abc6c721b01d service swift] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:04.869922 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f801536-5b05-4449-a70e-928140b22022 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dc226f7ddcb49ecacb51ce57fdfc11b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:04.870849 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f801536-5b05-4449-a70e-928140b22022 service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:04.871359 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f801536-5b05-4449-a70e-928140b22022 service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:04.871816 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f801536-5b05-4449-a70e-928140b22022 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:04.888162 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f801536-5b05-4449-a70e-928140b22022 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:04.890850 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f801536-5b05-4449-a70e-928140b22022 service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:04.917128 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1346/2690] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:42:04 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 80 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:42:07.723607 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fe23d71-0456-45ca-8149-76eae6da85ce service neutron] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:07.725215 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-2fe23d71-0456-45ca-8149-76eae6da85ce service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:07.760982 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fe23d71-0456-45ca-8149-76eae6da85ce service neutron] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:07.768815 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80f8eb9b-50fe-4136-a871-73b31990e88f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:07.770098 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80f8eb9b-50fe-4136-a871-73b31990e88f service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:07.770462 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80f8eb9b-50fe-4136-a871-73b31990e88f service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:07.770832 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80f8eb9b-50fe-4136-a871-73b31990e88f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:07.773414 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-80f8eb9b-50fe-4136-a871-73b31990e88f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:07.983463 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80f8eb9b-50fe-4136-a871-73b31990e88f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:07.985058 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80f8eb9b-50fe-4136-a871-73b31990e88f service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:08.003720 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1345/2691] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:42:07 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 281 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:42:08.156425 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f801536-5b05-4449-a70e-928140b22022 service cinder] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:08.157968 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-4f801536-5b05-4449-a70e-928140b22022 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:08.185477 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f801536-5b05-4449-a70e-928140b22022 service cinder] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:08.191993 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7d2ce2f-2570-45f2-ab03-f300181ea0db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:08.193006 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7d2ce2f-2570-45f2-ab03-f300181ea0db admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:08.193311 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7d2ce2f-2570-45f2-ab03-f300181ea0db admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:08.193626 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7d2ce2f-2570-45f2-ab03-f300181ea0db admin admin] PATH_INFO: `/v3/users/b6a3df8d796e4ae6ac358f103903d427` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:08.195386 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7d2ce2f-2570-45f2-ab03-f300181ea0db admin admin] RBAC: Authorizing `identity:delete_user(user_id=b6a3df8d796e4ae6ac358f103903d427)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:08.198510 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7d2ce2f-2570-45f2-ab03-f300181ea0db admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:08.269719 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-c7d2ce2f-2570-45f2-ab03-f300181ea0db admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b6a3df8d796e4ae6ac358f103903d427'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:08.291577 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-c7d2ce2f-2570-45f2-ab03-f300181ea0db admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b6a3df8d796e4ae6ac358f103903d427'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:08.293809 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-c7d2ce2f-2570-45f2-ab03-f300181ea0db admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b6a3df8d796e4ae6ac358f103903d427'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:08.295005 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-c7d2ce2f-2570-45f2-ab03-f300181ea0db admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b6a3df8d796e4ae6ac358f103903d427'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:08.311036 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-c7d2ce2f-2570-45f2-ab03-f300181ea0db admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b6a3df8d796e4ae6ac358f103903d427'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:08.332638 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1347/2692] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:42:08 2022] DELETE /identity/v3/users/b6a3df8d796e4ae6ac358f103903d427 => generated 0 bytes in 177 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:42:08.606639 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80f8eb9b-50fe-4136-a871-73b31990e88f service neutron] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:08.623671 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80f8eb9b-50fe-4136-a871-73b31990e88f service neutron] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:08.653506 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:42:08.741280 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de4a250e-c2e6-4af9-b16b-e450c7cd1eea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:08.742118 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de4a250e-c2e6-4af9-b16b-e450c7cd1eea admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:08.742508 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de4a250e-c2e6-4af9-b16b-e450c7cd1eea admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:08.742889 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de4a250e-c2e6-4af9-b16b-e450c7cd1eea admin admin] PATH_INFO: `/v3/projects/817ab3905c8541dc9c2bab51764984d0` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:08.744249 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de4a250e-c2e6-4af9-b16b-e450c7cd1eea admin admin] RBAC: Authorizing `identity:delete_project(project_id=817ab3905c8541dc9c2bab51764984d0)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:08.746482 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de4a250e-c2e6-4af9-b16b-e450c7cd1eea admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:08.822355 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-de4a250e-c2e6-4af9-b16b-e450c7cd1eea admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '817ab3905c8541dc9c2bab51764984d0', 'request_id': 'req-de4a250e-c2e6-4af9-b16b-e450c7cd1eea'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:08.829897 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-de4a250e-c2e6-4af9-b16b-e450c7cd1eea admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '817ab3905c8541dc9c2bab51764984d0', 'request_id': 'req-de4a250e-c2e6-4af9-b16b-e450c7cd1eea'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:08.838215 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-de4a250e-c2e6-4af9-b16b-e450c7cd1eea admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '817ab3905c8541dc9c2bab51764984d0', 'request_id': 'req-de4a250e-c2e6-4af9-b16b-e450c7cd1eea'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:08.856035 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-de4a250e-c2e6-4af9-b16b-e450c7cd1eea admin admin] The token cache is being invalidate because project 817ab3905c8541dc9c2bab51764984d0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81434) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:42:08.856723 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-de4a250e-c2e6-4af9-b16b-e450c7cd1eea admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:08.858063 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-de4a250e-c2e6-4af9-b16b-e450c7cd1eea admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:08.863075 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1346/2693] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:42:08 2022] DELETE /identity/v3/projects/817ab3905c8541dc9c2bab51764984d0 => generated 0 bytes in 255 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:42:08.865477 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac2b6094-bf6b-4ff5-9b2d-4be43facf73e None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:08.866226 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac2b6094-bf6b-4ff5-9b2d-4be43facf73e None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:08.866718 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac2b6094-bf6b-4ff5-9b2d-4be43facf73e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:08.888013 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-ac2b6094-bf6b-4ff5-9b2d-4be43facf73e None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:42:08.889415 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-ac2b6094-bf6b-4ff5-9b2d-4be43facf73e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:08.912285 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:42:08.980426 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1348/2694] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:42:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:42:08.982124 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e97e9bfb-5b4e-4d43-ae64-1efaa0836960 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:08.982433 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e97e9bfb-5b4e-4d43-ae64-1efaa0836960 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:08.982666 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e97e9bfb-5b4e-4d43-ae64-1efaa0836960 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:09.006883 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-e97e9bfb-5b4e-4d43-ae64-1efaa0836960 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:42:09.009046 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-e97e9bfb-5b4e-4d43-ae64-1efaa0836960 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:09.022329 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1347/2695] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:42:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:42:09.027312 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac2b6094-bf6b-4ff5-9b2d-4be43facf73e None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:09.041295 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac2b6094-bf6b-4ff5-9b2d-4be43facf73e None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:09.050443 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc7ba8d9-3148-48d9-b9f4-e6542845df6a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:09.051389 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc7ba8d9-3148-48d9-b9f4-e6542845df6a admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:09.051670 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc7ba8d9-3148-48d9-b9f4-e6542845df6a admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:09.052895 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc7ba8d9-3148-48d9-b9f4-e6542845df6a admin admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:09.053875 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc7ba8d9-3148-48d9-b9f4-e6542845df6a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:42:09.054359 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc7ba8d9-3148-48d9-b9f4-e6542845df6a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:09.056861 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc7ba8d9-3148-48d9-b9f4-e6542845df6a admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:09.073867 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1349/2696] 10.222.0.90 () {58 vars in 1153 bytes} [Mon Aug 29 21:42:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:42:09.079757 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e97e9bfb-5b4e-4d43-ae64-1efaa0836960 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:09.093233 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e97e9bfb-5b4e-4d43-ae64-1efaa0836960 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:09.101229 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7de5f94b-2cc1-4748-8ef1-38ad2f52f00a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:09.102124 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7de5f94b-2cc1-4748-8ef1-38ad2f52f00a admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:09.102416 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7de5f94b-2cc1-4748-8ef1-38ad2f52f00a admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:09.102692 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7de5f94b-2cc1-4748-8ef1-38ad2f52f00a admin admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:09.103348 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7de5f94b-2cc1-4748-8ef1-38ad2f52f00a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:09.105287 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7de5f94b-2cc1-4748-8ef1-38ad2f52f00a admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:09.161804 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1348/2697] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:42:09 2022] POST /identity/v3/projects => generated 379 bytes in 82 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:42:09.166165 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc7ba8d9-3148-48d9-b9f4-e6542845df6a admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:09.179890 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc7ba8d9-3148-48d9-b9f4-e6542845df6a admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:09.279615 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b2625dd-a2d0-4d4d-a661-af5bcd609528 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:09.280831 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b2625dd-a2d0-4d4d-a661-af5bcd609528 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:09.282106 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b2625dd-a2d0-4d4d-a661-af5bcd609528 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:09.283038 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b2625dd-a2d0-4d4d-a661-af5bcd609528 admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:09.284131 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b2625dd-a2d0-4d4d-a661-af5bcd609528 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:09.286803 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b2625dd-a2d0-4d4d-a661-af5bcd609528 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:09.333927 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1350/2698] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:42:09 2022] POST /identity/v3/users => generated 290 bytes in 164 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 21:42:09.335188 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7de5f94b-2cc1-4748-8ef1-38ad2f52f00a admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:09.361001 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7de5f94b-2cc1-4748-8ef1-38ad2f52f00a admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:09.366644 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d0ac49a-a07c-4e7b-818a-3fae60ef3054 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:09.367409 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d0ac49a-a07c-4e7b-818a-3fae60ef3054 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:09.368145 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d0ac49a-a07c-4e7b-818a-3fae60ef3054 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:09.368456 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d0ac49a-a07c-4e7b-818a-3fae60ef3054 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:09.369139 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d0ac49a-a07c-4e7b-818a-3fae60ef3054 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:09.371411 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d0ac49a-a07c-4e7b-818a-3fae60ef3054 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:09.389832 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1349/2699] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:42:09 2022] GET /identity/v3/roles => generated 1420 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 29 21:42:09.392845 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b2625dd-a2d0-4d4d-a661-af5bcd609528 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:09.424253 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b2625dd-a2d0-4d4d-a661-af5bcd609528 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:09.429268 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b893e0b-1dba-4596-9f97-f53ada2af5a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:09.429926 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b893e0b-1dba-4596-9f97-f53ada2af5a4 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:09.430195 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b893e0b-1dba-4596-9f97-f53ada2af5a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:09.430471 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b893e0b-1dba-4596-9f97-f53ada2af5a4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:09.431488 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b893e0b-1dba-4596-9f97-f53ada2af5a4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:09.433355 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b893e0b-1dba-4596-9f97-f53ada2af5a4 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:09.447173 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1351/2700] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:42:09 2022] GET /identity/v3/roles => generated 1420 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:42:09.451961 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d0ac49a-a07c-4e7b-818a-3fae60ef3054 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:09.468210 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d0ac49a-a07c-4e7b-818a-3fae60ef3054 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:09.480056 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-575368bc-ae27-48ea-87ed-daabe3e52e0c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:09.480691 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-575368bc-ae27-48ea-87ed-daabe3e52e0c admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:09.481097 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-575368bc-ae27-48ea-87ed-daabe3e52e0c admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:09.481387 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-575368bc-ae27-48ea-87ed-daabe3e52e0c admin admin] PATH_INFO: `/v3/projects/1bf1b8ef932d4c78828ec664181239c9/users/58cf56ee4cbc4f84a10fbf6b3d33e083/roles/02e242d5520740d4b77a5f4008c7eb57` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:09.558074 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-575368bc-ae27-48ea-87ed-daabe3e52e0c admin admin] RBAC: Authorizing `identity:create_grant(project_id=1bf1b8ef932d4c78828ec664181239c9, user_id=58cf56ee4cbc4f84a10fbf6b3d33e083, role_id=02e242d5520740d4b77a5f4008c7eb57)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:09.561557 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-575368bc-ae27-48ea-87ed-daabe3e52e0c admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:09.614604 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1350/2701] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:42:09 2022] PUT /identity/v3/projects/1bf1b8ef932d4c78828ec664181239c9/users/58cf56ee4cbc4f84a10fbf6b3d33e083/roles/02e242d5520740d4b77a5f4008c7eb57 => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:42:09.617628 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4dfb1b1-15c1-48a6-a6b0-8b83b32cac81 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:09.618175 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4dfb1b1-15c1-48a6-a6b0-8b83b32cac81 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:09.618669 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4dfb1b1-15c1-48a6-a6b0-8b83b32cac81 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:09.678354 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-f4dfb1b1-15c1-48a6-a6b0-8b83b32cac81 None None] MFA Rules not processed for user `58cf56ee4cbc4f84a10fbf6b3d33e083`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:42:09.680031 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-f4dfb1b1-15c1-48a6-a6b0-8b83b32cac81 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:09.866147 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1352/2702] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:42:09 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 250 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:42:09.870835 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-575368bc-ae27-48ea-87ed-daabe3e52e0c admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:09.983066 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:42:10.038288 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-575368bc-ae27-48ea-87ed-daabe3e52e0c admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:10.053299 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44ef2dec-b382-4e69-8452-e4cb23bda050 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:10.054426 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44ef2dec-b382-4e69-8452-e4cb23bda050 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:10.054917 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44ef2dec-b382-4e69-8452-e4cb23bda050 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:10.055345 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44ef2dec-b382-4e69-8452-e4cb23bda050 admin admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:10.056315 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44ef2dec-b382-4e69-8452-e4cb23bda050 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:10.060629 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44ef2dec-b382-4e69-8452-e4cb23bda050 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:10.112382 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1351/2703] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:42:09 2022] POST /identity/v3/projects => generated 379 bytes in 242 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:42:10.113727 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4dfb1b1-15c1-48a6-a6b0-8b83b32cac81 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:10.131563 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4dfb1b1-15c1-48a6-a6b0-8b83b32cac81 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:10.157523 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:42:10.209176 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bd15ec7-8ac1-4272-b69a-865000294662 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:10.210625 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bd15ec7-8ac1-4272-b69a-865000294662 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:10.210907 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bd15ec7-8ac1-4272-b69a-865000294662 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:10.211181 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bd15ec7-8ac1-4272-b69a-865000294662 admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:10.211815 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bd15ec7-8ac1-4272-b69a-865000294662 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:10.213744 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bd15ec7-8ac1-4272-b69a-865000294662 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:10.240554 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1353/2704] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:42:10 2022] POST /identity/v3/users => generated 296 bytes in 128 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 21:42:10.244327 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44ef2dec-b382-4e69-8452-e4cb23bda050 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:10.257813 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44ef2dec-b382-4e69-8452-e4cb23bda050 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:10.266176 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45d0231c-d7e3-4f79-a171-8076808e5c5f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:10.267640 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45d0231c-d7e3-4f79-a171-8076808e5c5f admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:10.268220 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45d0231c-d7e3-4f79-a171-8076808e5c5f admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:10.268721 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45d0231c-d7e3-4f79-a171-8076808e5c5f admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:10.269705 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45d0231c-d7e3-4f79-a171-8076808e5c5f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:10.271601 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45d0231c-d7e3-4f79-a171-8076808e5c5f admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:10.289763 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1352/2705] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:42:10 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:42:10.294730 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bd15ec7-8ac1-4272-b69a-865000294662 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:10.306937 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bd15ec7-8ac1-4272-b69a-865000294662 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:10.313195 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ccbf890-b245-4644-b71f-593e6cbf6516 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:10.314206 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ccbf890-b245-4644-b71f-593e6cbf6516 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:10.314637 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ccbf890-b245-4644-b71f-593e6cbf6516 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:10.315091 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ccbf890-b245-4644-b71f-593e6cbf6516 admin admin] PATH_INFO: `/v3/projects/57208a72edb44fc0a76a54748420feca/users/b2b9097988284575ae365c2d18ae2cfa/roles/a6e562cdc96c4be691d638990f8403fd` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:10.338487 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ccbf890-b245-4644-b71f-593e6cbf6516 admin admin] RBAC: Authorizing `identity:create_grant(project_id=57208a72edb44fc0a76a54748420feca, user_id=b2b9097988284575ae365c2d18ae2cfa, role_id=a6e562cdc96c4be691d638990f8403fd)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:10.341219 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ccbf890-b245-4644-b71f-593e6cbf6516 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:10.359645 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1354/2706] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:42:10 2022] PUT /identity/v3/projects/57208a72edb44fc0a76a54748420feca/users/b2b9097988284575ae365c2d18ae2cfa/roles/a6e562cdc96c4be691d638990f8403fd => generated 0 bytes in 65 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:42:10.363601 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45d0231c-d7e3-4f79-a171-8076808e5c5f admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:10.379686 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45d0231c-d7e3-4f79-a171-8076808e5c5f admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:10.384967 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ccbf890-b245-4644-b71f-593e6cbf6516 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:10.386774 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-8ccbf890-b245-4644-b71f-593e6cbf6516 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:10.495005 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16614c38-df13-436e-9c0e-08905b90ebfd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:10.497051 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16614c38-df13-436e-9c0e-08905b90ebfd admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:10.498107 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16614c38-df13-436e-9c0e-08905b90ebfd admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:10.498515 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16614c38-df13-436e-9c0e-08905b90ebfd admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:10.499256 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16614c38-df13-436e-9c0e-08905b90ebfd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:10.501940 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16614c38-df13-436e-9c0e-08905b90ebfd admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:10.503678 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ccbf890-b245-4644-b71f-593e6cbf6516 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:10.514926 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b233d5c0-1ac0-4bd1-966e-533b9501ed25 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:10.517300 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b233d5c0-1ac0-4bd1-966e-533b9501ed25 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:10.517656 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b233d5c0-1ac0-4bd1-966e-533b9501ed25 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:10.518019 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b233d5c0-1ac0-4bd1-966e-533b9501ed25 admin admin] PATH_INFO: `/v3/users/cd8e9c9275144a339a18808375ba8e33` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:10.520683 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1353/2707] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:42:10 2022] GET /identity/v3/roles => generated 1420 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:42:10.521868 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b233d5c0-1ac0-4bd1-966e-533b9501ed25 admin admin] RBAC: Authorizing `identity:delete_user(user_id=cd8e9c9275144a339a18808375ba8e33)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:10.522015 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16614c38-df13-436e-9c0e-08905b90ebfd admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:10.529068 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b233d5c0-1ac0-4bd1-966e-533b9501ed25 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:10.539244 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16614c38-df13-436e-9c0e-08905b90ebfd admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:10.547489 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1a53704-a6d3-4b05-b746-e7d9e40cce68 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:10.548767 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a53704-a6d3-4b05-b746-e7d9e40cce68 admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:10.549047 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a53704-a6d3-4b05-b746-e7d9e40cce68 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:10.550046 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a53704-a6d3-4b05-b746-e7d9e40cce68 admin admin] PATH_INFO: `/v3/projects/57208a72edb44fc0a76a54748420feca/users/b2b9097988284575ae365c2d18ae2cfa/roles/a6e562cdc96c4be691d638990f8403fd` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:10.553611 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1a53704-a6d3-4b05-b746-e7d9e40cce68 admin admin] RBAC: Authorizing `identity:create_grant(project_id=57208a72edb44fc0a76a54748420feca, user_id=b2b9097988284575ae365c2d18ae2cfa, role_id=a6e562cdc96c4be691d638990f8403fd)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:10.562023 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1a53704-a6d3-4b05-b746-e7d9e40cce68 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:10.606989 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1354/2708] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:42:10 2022] PUT /identity/v3/projects/57208a72edb44fc0a76a54748420feca/users/b2b9097988284575ae365c2d18ae2cfa/roles/a6e562cdc96c4be691d638990f8403fd => generated 0 bytes in 83 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:42:10.617753 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f3b71d6-5d97-41e0-8354-8631be019d20 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:10.618097 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f3b71d6-5d97-41e0-8354-8631be019d20 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:10.618441 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f3b71d6-5d97-41e0-8354-8631be019d20 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:10.665691 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-b233d5c0-1ac0-4bd1-966e-533b9501ed25 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cd8e9c9275144a339a18808375ba8e33'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:10.681089 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-b233d5c0-1ac0-4bd1-966e-533b9501ed25 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cd8e9c9275144a339a18808375ba8e33'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:10.681840 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-b233d5c0-1ac0-4bd1-966e-533b9501ed25 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cd8e9c9275144a339a18808375ba8e33'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:10.682476 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-b233d5c0-1ac0-4bd1-966e-533b9501ed25 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cd8e9c9275144a339a18808375ba8e33'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:10.703687 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-b233d5c0-1ac0-4bd1-966e-533b9501ed25 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cd8e9c9275144a339a18808375ba8e33'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:10.712748 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-0f3b71d6-5d97-41e0-8354-8631be019d20 None None] MFA Rules not processed for user `b2b9097988284575ae365c2d18ae2cfa`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:42:10.714981 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-0f3b71d6-5d97-41e0-8354-8631be019d20 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:10.729011 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1355/2709] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:42:10 2022] DELETE /identity/v3/users/cd8e9c9275144a339a18808375ba8e33 => generated 0 bytes in 345 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 21:42:10.748284 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3690cfef-409c-4967-8d28-12f2c514c975 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:10.748717 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3690cfef-409c-4967-8d28-12f2c514c975 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:10.749147 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3690cfef-409c-4967-8d28-12f2c514c975 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:10.774172 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=85,keystone:INSERT=5,keystone:DELETE=10 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:42:10.784164 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-3690cfef-409c-4967-8d28-12f2c514c975 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:42:10.786201 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-3690cfef-409c-4967-8d28-12f2c514c975 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:10.837425 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1355/2710] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:42:10 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 221 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:42:10.862259 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f3b71d6-5d97-41e0-8354-8631be019d20 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:10.864265 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-0f3b71d6-5d97-41e0-8354-8631be019d20 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:10.920733 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:42:11.021863 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1356/2711] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:42:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 287 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:42:11.046089 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f3b71d6-5d97-41e0-8354-8631be019d20 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:11.050265 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3690cfef-409c-4967-8d28-12f2c514c975 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:11.051824 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-3690cfef-409c-4967-8d28-12f2c514c975 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:11.148512 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d561baf-8157-4e64-bf50-4dff875832a5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dc226f7ddcb49ecacb51ce57fdfc11b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:11.149136 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d561baf-8157-4e64-bf50-4dff875832a5 service cinder] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:11.149417 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d561baf-8157-4e64-bf50-4dff875832a5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:11.149690 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d561baf-8157-4e64-bf50-4dff875832a5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:11.176098 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3690cfef-409c-4967-8d28-12f2c514c975 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:11.180217 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fe02856-1abc-407f-ab93-a4ffef9ce406 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:11.181255 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fe02856-1abc-407f-ab93-a4ffef9ce406 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:11.181736 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fe02856-1abc-407f-ab93-a4ffef9ce406 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:11.182237 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fe02856-1abc-407f-ab93-a4ffef9ce406 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:11.200279 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fe02856-1abc-407f-ab93-a4ffef9ce406 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:11.201624 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fe02856-1abc-407f-ab93-a4ffef9ce406 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:11.213010 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1357/2712] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:42:11 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 163 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 21:42:11.257518 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d561baf-8157-4e64-bf50-4dff875832a5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:11.260019 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d561baf-8157-4e64-bf50-4dff875832a5 service cinder] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:11.287353 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1356/2713] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:42:10 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 425 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:42:11.421364 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fe02856-1abc-407f-ab93-a4ffef9ce406 service neutron] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:11.436793 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fe02856-1abc-407f-ab93-a4ffef9ce406 service neutron] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:11.443000 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39733eb5-d249-4a8f-ba28-d78a4e8b8555 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dc226f7ddcb49ecacb51ce57fdfc11b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:11.443608 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39733eb5-d249-4a8f-ba28-d78a4e8b8555 service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:11.444579 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39733eb5-d249-4a8f-ba28-d78a4e8b8555 service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:11.444854 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39733eb5-d249-4a8f-ba28-d78a4e8b8555 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:11.446712 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-39733eb5-d249-4a8f-ba28-d78a4e8b8555 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:11.574974 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39733eb5-d249-4a8f-ba28-d78a4e8b8555 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:11.578110 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39733eb5-d249-4a8f-ba28-d78a4e8b8555 service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:11.610748 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1358/2714] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:42:11 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 190 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:42:11.659148 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d561baf-8157-4e64-bf50-4dff875832a5 service cinder] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:11.660753 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-8d561baf-8157-4e64-bf50-4dff875832a5 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:11.694195 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d561baf-8157-4e64-bf50-4dff875832a5 service cinder] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:11.706813 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a89e465-496d-4650-b1f5-513baa59285f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:11.708054 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a89e465-496d-4650-b1f5-513baa59285f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:11.708054 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a89e465-496d-4650-b1f5-513baa59285f admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:11.708054 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a89e465-496d-4650-b1f5-513baa59285f admin admin] PATH_INFO: `/v3/projects/38ffe55e1bf34a21af1add5945c01761` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:11.709389 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a89e465-496d-4650-b1f5-513baa59285f admin admin] RBAC: Authorizing `identity:delete_project(project_id=38ffe55e1bf34a21af1add5945c01761)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:11.721956 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a89e465-496d-4650-b1f5-513baa59285f admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:11.847753 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-9a89e465-496d-4650-b1f5-513baa59285f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '38ffe55e1bf34a21af1add5945c01761', 'request_id': 'req-9a89e465-496d-4650-b1f5-513baa59285f'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:11.869778 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-9a89e465-496d-4650-b1f5-513baa59285f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '38ffe55e1bf34a21af1add5945c01761', 'request_id': 'req-9a89e465-496d-4650-b1f5-513baa59285f'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:11.882823 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-9a89e465-496d-4650-b1f5-513baa59285f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '38ffe55e1bf34a21af1add5945c01761', 'request_id': 'req-9a89e465-496d-4650-b1f5-513baa59285f'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:11.913374 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-9a89e465-496d-4650-b1f5-513baa59285f admin admin] The token cache is being invalidate because project 38ffe55e1bf34a21af1add5945c01761 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81434) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:42:11.913958 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-9a89e465-496d-4650-b1f5-513baa59285f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:11.915156 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-9a89e465-496d-4650-b1f5-513baa59285f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:11.917021 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1357/2715] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:42:11 2022] DELETE /identity/v3/projects/38ffe55e1bf34a21af1add5945c01761 => generated 0 bytes in 258 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:42:11.921246 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39733eb5-d249-4a8f-ba28-d78a4e8b8555 service cinder] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:11.939203 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39733eb5-d249-4a8f-ba28-d78a4e8b8555 service cinder] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:12.022131 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:42:12.052532 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b90feaed-3c39-4cf2-b240-69ed5c827e52 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:12.053592 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b90feaed-3c39-4cf2-b240-69ed5c827e52 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:12.053927 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b90feaed-3c39-4cf2-b240-69ed5c827e52 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:12.054371 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b90feaed-3c39-4cf2-b240-69ed5c827e52 admin admin] PATH_INFO: `/v3/users/beb3c60c4f1b414dbbbd99d5e76050b5` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:12.055537 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b90feaed-3c39-4cf2-b240-69ed5c827e52 admin admin] RBAC: Authorizing `identity:delete_user(user_id=beb3c60c4f1b414dbbbd99d5e76050b5)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:12.057859 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b90feaed-3c39-4cf2-b240-69ed5c827e52 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:12.150489 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-b90feaed-3c39-4cf2-b240-69ed5c827e52 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'beb3c60c4f1b414dbbbd99d5e76050b5'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:12.171534 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-b90feaed-3c39-4cf2-b240-69ed5c827e52 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'beb3c60c4f1b414dbbbd99d5e76050b5'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:12.172407 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-b90feaed-3c39-4cf2-b240-69ed5c827e52 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'beb3c60c4f1b414dbbbd99d5e76050b5'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:12.173112 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-b90feaed-3c39-4cf2-b240-69ed5c827e52 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'beb3c60c4f1b414dbbbd99d5e76050b5'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:12.184211 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-b90feaed-3c39-4cf2-b240-69ed5c827e52 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'beb3c60c4f1b414dbbbd99d5e76050b5'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:12.219215 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1359/2716] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:42:11 2022] DELETE /identity/v3/users/beb3c60c4f1b414dbbbd99d5e76050b5 => generated 0 bytes in 284 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:42:12.816974 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a89e465-496d-4650-b1f5-513baa59285f admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:12.819376 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-9a89e465-496d-4650-b1f5-513baa59285f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:12.839883 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a89e465-496d-4650-b1f5-513baa59285f admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:12.861890 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:42:12.974232 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-890806fb-6ba2-420a-bba7-60b6c7eb7269 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:12.975461 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-890806fb-6ba2-420a-bba7-60b6c7eb7269 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:12.975746 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-890806fb-6ba2-420a-bba7-60b6c7eb7269 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:12.976020 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-890806fb-6ba2-420a-bba7-60b6c7eb7269 admin admin] PATH_INFO: `/v3/projects/2e420d2c67c94d97b369b2c8f8e26530` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:12.977298 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-890806fb-6ba2-420a-bba7-60b6c7eb7269 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2e420d2c67c94d97b369b2c8f8e26530)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:12.979615 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-890806fb-6ba2-420a-bba7-60b6c7eb7269 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:13.065210 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-890806fb-6ba2-420a-bba7-60b6c7eb7269 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2e420d2c67c94d97b369b2c8f8e26530', 'request_id': 'req-890806fb-6ba2-420a-bba7-60b6c7eb7269'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:13.087939 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-890806fb-6ba2-420a-bba7-60b6c7eb7269 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2e420d2c67c94d97b369b2c8f8e26530', 'request_id': 'req-890806fb-6ba2-420a-bba7-60b6c7eb7269'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:13.106651 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-890806fb-6ba2-420a-bba7-60b6c7eb7269 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2e420d2c67c94d97b369b2c8f8e26530', 'request_id': 'req-890806fb-6ba2-420a-bba7-60b6c7eb7269'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:13.149311 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-890806fb-6ba2-420a-bba7-60b6c7eb7269 admin admin] The token cache is being invalidate because project 2e420d2c67c94d97b369b2c8f8e26530 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81434) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:42:13.149859 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-890806fb-6ba2-420a-bba7-60b6c7eb7269 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:13.155180 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-890806fb-6ba2-420a-bba7-60b6c7eb7269 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:13.159609 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1358/2717] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:42:12 2022] DELETE /identity/v3/projects/2e420d2c67c94d97b369b2c8f8e26530 => generated 0 bytes in 343 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:42:13.168656 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfe45b69-8eaf-47cd-9f0d-cfa2651d3d1c None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:13.169506 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfe45b69-8eaf-47cd-9f0d-cfa2651d3d1c None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:13.169996 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfe45b69-8eaf-47cd-9f0d-cfa2651d3d1c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:13.216496 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-cfe45b69-8eaf-47cd-9f0d-cfa2651d3d1c None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:42:13.217948 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-cfe45b69-8eaf-47cd-9f0d-cfa2651d3d1c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:13.352296 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1360/2718] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:42:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:42:13.365957 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da768d45-47ca-41d4-8062-14f4124240b5 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:13.366433 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da768d45-47ca-41d4-8062-14f4124240b5 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:13.366791 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da768d45-47ca-41d4-8062-14f4124240b5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:13.398442 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-da768d45-47ca-41d4-8062-14f4124240b5 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:42:13.399784 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-da768d45-47ca-41d4-8062-14f4124240b5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:13.416658 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1359/2719] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:42:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 51 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:42:13.417245 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfe45b69-8eaf-47cd-9f0d-cfa2651d3d1c None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:13.434918 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfe45b69-8eaf-47cd-9f0d-cfa2651d3d1c None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:13.442089 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21b9ea25-da83-4366-9e46-de2c1cacc445 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:13.442770 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21b9ea25-da83-4366-9e46-de2c1cacc445 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:13.443113 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21b9ea25-da83-4366-9e46-de2c1cacc445 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:13.443454 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21b9ea25-da83-4366-9e46-de2c1cacc445 admin admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:13.444192 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21b9ea25-da83-4366-9e46-de2c1cacc445 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:42:13.444591 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21b9ea25-da83-4366-9e46-de2c1cacc445 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:13.446442 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21b9ea25-da83-4366-9e46-de2c1cacc445 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:13.477276 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1361/2720] 10.222.0.90 () {58 vars in 1153 bytes} [Mon Aug 29 21:42:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 61 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:42:13.483021 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da768d45-47ca-41d4-8062-14f4124240b5 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:13.496018 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da768d45-47ca-41d4-8062-14f4124240b5 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:13.517411 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f8cfef1-e1e3-49bd-b539-0976f9c08362 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:13.518141 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f8cfef1-e1e3-49bd-b539-0976f9c08362 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:13.518429 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f8cfef1-e1e3-49bd-b539-0976f9c08362 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:13.518725 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f8cfef1-e1e3-49bd-b539-0976f9c08362 admin admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:13.519403 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f8cfef1-e1e3-49bd-b539-0976f9c08362 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:13.521432 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f8cfef1-e1e3-49bd-b539-0976f9c08362 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:13.564214 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1360/2721] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:42:13 2022] POST /identity/v3/projects => generated 375 bytes in 79 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:42:13.565449 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21b9ea25-da83-4366-9e46-de2c1cacc445 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:13.593229 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21b9ea25-da83-4366-9e46-de2c1cacc445 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:13.692844 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:42:13.710158 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f99e9fa4-855e-4fc6-be6f-4c0c709cbc60 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:13.711376 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f99e9fa4-855e-4fc6-be6f-4c0c709cbc60 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:13.711680 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f99e9fa4-855e-4fc6-be6f-4c0c709cbc60 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:13.712012 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f99e9fa4-855e-4fc6-be6f-4c0c709cbc60 admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:13.712904 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f99e9fa4-855e-4fc6-be6f-4c0c709cbc60 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:13.715678 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f99e9fa4-855e-4fc6-be6f-4c0c709cbc60 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:13.747452 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1362/2722] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:42:13 2022] POST /identity/v3/users => generated 288 bytes in 182 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:42:13.752286 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f8cfef1-e1e3-49bd-b539-0976f9c08362 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:13.780197 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f8cfef1-e1e3-49bd-b539-0976f9c08362 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:13.788993 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5738532-5bdc-411e-95d2-bb811f6603cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:13.789667 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5738532-5bdc-411e-95d2-bb811f6603cd admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:13.790043 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5738532-5bdc-411e-95d2-bb811f6603cd admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:13.790210 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5738532-5bdc-411e-95d2-bb811f6603cd admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:13.806961 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5738532-5bdc-411e-95d2-bb811f6603cd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:13.808942 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5738532-5bdc-411e-95d2-bb811f6603cd admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:13.834711 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1361/2723] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:42:13 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:42:13.839858 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f99e9fa4-855e-4fc6-be6f-4c0c709cbc60 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:13.853913 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f99e9fa4-855e-4fc6-be6f-4c0c709cbc60 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:13.874074 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fe7941c-b50c-4cff-908b-798a705c835d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:13.875252 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe7941c-b50c-4cff-908b-798a705c835d admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:13.875516 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe7941c-b50c-4cff-908b-798a705c835d admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:13.875985 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe7941c-b50c-4cff-908b-798a705c835d admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:13.877050 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fe7941c-b50c-4cff-908b-798a705c835d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:13.879382 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fe7941c-b50c-4cff-908b-798a705c835d admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:13.900900 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1363/2724] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:42:13 2022] GET /identity/v3/roles => generated 1420 bytes in 62 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:42:13.906608 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5738532-5bdc-411e-95d2-bb811f6603cd admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:13.924193 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5738532-5bdc-411e-95d2-bb811f6603cd admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:13.934770 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c646a6e9-57e0-4949-8945-ecc172733a92 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:13.935708 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c646a6e9-57e0-4949-8945-ecc172733a92 admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:13.936174 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c646a6e9-57e0-4949-8945-ecc172733a92 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:13.936523 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c646a6e9-57e0-4949-8945-ecc172733a92 admin admin] PATH_INFO: `/v3/projects/5f7b23fc68b7490a9bf7e504704b541a/users/862248249d7b408eb5cd18d73733a32c/roles/02e242d5520740d4b77a5f4008c7eb57` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:13.958059 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c646a6e9-57e0-4949-8945-ecc172733a92 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5f7b23fc68b7490a9bf7e504704b541a, user_id=862248249d7b408eb5cd18d73733a32c, role_id=02e242d5520740d4b77a5f4008c7eb57)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:13.961296 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c646a6e9-57e0-4949-8945-ecc172733a92 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:13.984651 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1362/2725] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:42:13 2022] PUT /identity/v3/projects/5f7b23fc68b7490a9bf7e504704b541a/users/862248249d7b408eb5cd18d73733a32c/roles/02e242d5520740d4b77a5f4008c7eb57 => generated 0 bytes in 80 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:42:13.998005 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4009a844-c07f-4b88-832f-3d2237a741f9 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:13.998339 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4009a844-c07f-4b88-832f-3d2237a741f9 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:13.998738 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4009a844-c07f-4b88-832f-3d2237a741f9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:14.036865 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-4009a844-c07f-4b88-832f-3d2237a741f9 None None] MFA Rules not processed for user `862248249d7b408eb5cd18d73733a32c`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:42:14.038474 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-4009a844-c07f-4b88-832f-3d2237a741f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:14.132506 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1364/2726] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:42:13 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:42:14.144067 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c646a6e9-57e0-4949-8945-ecc172733a92 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:14.150596 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-c646a6e9-57e0-4949-8945-ecc172733a92 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:14.216275 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:42:14.239603 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c646a6e9-57e0-4949-8945-ecc172733a92 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:14.243455 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3211748-1941-49a2-b1f5-e52f521bc6c0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dc226f7ddcb49ecacb51ce57fdfc11b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:14.244172 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3211748-1941-49a2-b1f5-e52f521bc6c0 service cinder] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:14.244366 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3211748-1941-49a2-b1f5-e52f521bc6c0 service cinder] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:14.244662 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3211748-1941-49a2-b1f5-e52f521bc6c0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:14.256003 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3211748-1941-49a2-b1f5-e52f521bc6c0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:14.257486 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3211748-1941-49a2-b1f5-e52f521bc6c0 service cinder] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:14.268571 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1363/2727] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:42:14 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 125 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:42:15.274335 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4009a844-c07f-4b88-832f-3d2237a741f9 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:15.278999 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-4009a844-c07f-4b88-832f-3d2237a741f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:15.349409 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4009a844-c07f-4b88-832f-3d2237a741f9 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:15.353162 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b816a9c3-10d3-4da3-8c44-144b045420c1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:15.353822 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b816a9c3-10d3-4da3-8c44-144b045420c1 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:15.354197 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b816a9c3-10d3-4da3-8c44-144b045420c1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:15.354378 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b816a9c3-10d3-4da3-8c44-144b045420c1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:15.356137 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-b816a9c3-10d3-4da3-8c44-144b045420c1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:15.425813 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b816a9c3-10d3-4da3-8c44-144b045420c1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:15.427426 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b816a9c3-10d3-4da3-8c44-144b045420c1 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:15.438681 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1365/2728] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:42:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:42:21.987264 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3211748-1941-49a2-b1f5-e52f521bc6c0 service cinder] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:21.995235 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-e3211748-1941-49a2-b1f5-e52f521bc6c0 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:22.012124 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3211748-1941-49a2-b1f5-e52f521bc6c0 service cinder] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:22.038973 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aff966a-3ad1-4a7a-ae3b-87b42affb5bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:22.040251 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aff966a-3ad1-4a7a-ae3b-87b42affb5bc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:22.040884 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aff966a-3ad1-4a7a-ae3b-87b42affb5bc admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:22.041467 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aff966a-3ad1-4a7a-ae3b-87b42affb5bc admin admin] PATH_INFO: `/v3/users/58cf56ee4cbc4f84a10fbf6b3d33e083` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:22.052255 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aff966a-3ad1-4a7a-ae3b-87b42affb5bc admin admin] RBAC: Authorizing `identity:delete_user(user_id=58cf56ee4cbc4f84a10fbf6b3d33e083)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:22.056136 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aff966a-3ad1-4a7a-ae3b-87b42affb5bc admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:22.196244 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-9aff966a-3ad1-4a7a-ae3b-87b42affb5bc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '58cf56ee4cbc4f84a10fbf6b3d33e083'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:22.236904 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-9aff966a-3ad1-4a7a-ae3b-87b42affb5bc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '58cf56ee4cbc4f84a10fbf6b3d33e083'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:22.255151 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-9aff966a-3ad1-4a7a-ae3b-87b42affb5bc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '58cf56ee4cbc4f84a10fbf6b3d33e083'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:22.258885 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-9aff966a-3ad1-4a7a-ae3b-87b42affb5bc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '58cf56ee4cbc4f84a10fbf6b3d33e083'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:22.265713 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-9aff966a-3ad1-4a7a-ae3b-87b42affb5bc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '58cf56ee4cbc4f84a10fbf6b3d33e083'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:22.343702 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1364/2729] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:42:21 2022] DELETE /identity/v3/users/58cf56ee4cbc4f84a10fbf6b3d33e083 => generated 0 bytes in 357 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:42:22.349246 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-031b2c2c-468e-421b-b8c8-3027a3856ee4 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:22.349770 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-031b2c2c-468e-421b-b8c8-3027a3856ee4 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:22.349923 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-031b2c2c-468e-421b-b8c8-3027a3856ee4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:22.403955 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-031b2c2c-468e-421b-b8c8-3027a3856ee4 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:42:22.405422 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-031b2c2c-468e-421b-b8c8-3027a3856ee4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:22.413984 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:42:22.538012 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1366/2730] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:42:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 190 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:42:22.547282 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aff966a-3ad1-4a7a-ae3b-87b42affb5bc admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:22.548818 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-9aff966a-3ad1-4a7a-ae3b-87b42affb5bc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:22.779677 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aff966a-3ad1-4a7a-ae3b-87b42affb5bc admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:22.788136 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18947f3f-c530-441c-82b6-5cce053093ad service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:22.789417 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18947f3f-c530-441c-82b6-5cce053093ad service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:22.789713 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18947f3f-c530-441c-82b6-5cce053093ad service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:22.792491 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18947f3f-c530-441c-82b6-5cce053093ad service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:22.831537 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18947f3f-c530-441c-82b6-5cce053093ad service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:22.840056 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18947f3f-c530-441c-82b6-5cce053093ad service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:22.885052 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1365/2731] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:42:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 339 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:42:23.562713 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-031b2c2c-468e-421b-b8c8-3027a3856ee4 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:23.565405 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-031b2c2c-468e-421b-b8c8-3027a3856ee4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:23.597510 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-031b2c2c-468e-421b-b8c8-3027a3856ee4 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:23.604306 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e7fb50f-ea25-40f9-a295-5e1942538861 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:23.606640 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e7fb50f-ea25-40f9-a295-5e1942538861 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:23.607563 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e7fb50f-ea25-40f9-a295-5e1942538861 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:23.608041 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e7fb50f-ea25-40f9-a295-5e1942538861 admin admin] PATH_INFO: `/v3/projects/1bf1b8ef932d4c78828ec664181239c9` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:23.613094 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e7fb50f-ea25-40f9-a295-5e1942538861 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1bf1b8ef932d4c78828ec664181239c9)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:23.616815 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e7fb50f-ea25-40f9-a295-5e1942538861 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:23.709221 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-5e7fb50f-ea25-40f9-a295-5e1942538861 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1bf1b8ef932d4c78828ec664181239c9', 'request_id': 'req-5e7fb50f-ea25-40f9-a295-5e1942538861'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:23.724846 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-5e7fb50f-ea25-40f9-a295-5e1942538861 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1bf1b8ef932d4c78828ec664181239c9', 'request_id': 'req-5e7fb50f-ea25-40f9-a295-5e1942538861'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:23.739711 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-5e7fb50f-ea25-40f9-a295-5e1942538861 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1bf1b8ef932d4c78828ec664181239c9', 'request_id': 'req-5e7fb50f-ea25-40f9-a295-5e1942538861'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:23.744018 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18947f3f-c530-441c-82b6-5cce053093ad service neutron] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:23.745478 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-18947f3f-c530-441c-82b6-5cce053093ad service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:23.775237 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18947f3f-c530-441c-82b6-5cce053093ad service neutron] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:23.778767 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-5e7fb50f-ea25-40f9-a295-5e1942538861 admin admin] The token cache is being invalidate because project 1bf1b8ef932d4c78828ec664181239c9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:42:23.779174 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-5e7fb50f-ea25-40f9-a295-5e1942538861 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:23.781533 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-5e7fb50f-ea25-40f9-a295-5e1942538861 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:23.783729 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1367/2732] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:42:23 2022] DELETE /identity/v3/projects/1bf1b8ef932d4c78828ec664181239c9 => generated 0 bytes in 222 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:42:23.790857 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e7fb50f-ea25-40f9-a295-5e1942538861 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:23.798938 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-5e7fb50f-ea25-40f9-a295-5e1942538861 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:23.861816 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:42:23.873608 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62be350e-7624-4cd8-8fdc-a3fae4fffed4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:23.874948 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62be350e-7624-4cd8-8fdc-a3fae4fffed4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:23.875535 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62be350e-7624-4cd8-8fdc-a3fae4fffed4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:23.875993 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62be350e-7624-4cd8-8fdc-a3fae4fffed4 admin admin] PATH_INFO: `/v3/users/3376818708e443928951fd5f517c9d83` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:23.877618 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62be350e-7624-4cd8-8fdc-a3fae4fffed4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3376818708e443928951fd5f517c9d83)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:23.880748 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62be350e-7624-4cd8-8fdc-a3fae4fffed4 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:23.939350 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e7fb50f-ea25-40f9-a295-5e1942538861 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:23.953321 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13b69663-73c6-4076-be3e-5760b4ac9b01 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:23.953321 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b69663-73c6-4076-be3e-5760b4ac9b01 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:23.956100 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b69663-73c6-4076-be3e-5760b4ac9b01 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:23.956100 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b69663-73c6-4076-be3e-5760b4ac9b01 admin admin] PATH_INFO: `/v3/users/b2b9097988284575ae365c2d18ae2cfa` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:23.962104 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13b69663-73c6-4076-be3e-5760b4ac9b01 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b2b9097988284575ae365c2d18ae2cfa)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:23.963281 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-62be350e-7624-4cd8-8fdc-a3fae4fffed4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3376818708e443928951fd5f517c9d83'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:23.964039 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13b69663-73c6-4076-be3e-5760b4ac9b01 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:23.974734 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-62be350e-7624-4cd8-8fdc-a3fae4fffed4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3376818708e443928951fd5f517c9d83'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:23.985437 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-62be350e-7624-4cd8-8fdc-a3fae4fffed4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3376818708e443928951fd5f517c9d83'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:23.986109 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-62be350e-7624-4cd8-8fdc-a3fae4fffed4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3376818708e443928951fd5f517c9d83'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:23.986794 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-62be350e-7624-4cd8-8fdc-a3fae4fffed4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3376818708e443928951fd5f517c9d83'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:24.015326 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1366/2733] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:42:23 2022] DELETE /identity/v3/users/3376818708e443928951fd5f517c9d83 => generated 0 bytes in 272 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 21:42:24.041733 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:42:24.061130 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-13b69663-73c6-4076-be3e-5760b4ac9b01 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b2b9097988284575ae365c2d18ae2cfa'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:24.082319 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-13b69663-73c6-4076-be3e-5760b4ac9b01 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b2b9097988284575ae365c2d18ae2cfa'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:24.083306 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-13b69663-73c6-4076-be3e-5760b4ac9b01 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b2b9097988284575ae365c2d18ae2cfa'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:24.084085 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-13b69663-73c6-4076-be3e-5760b4ac9b01 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b2b9097988284575ae365c2d18ae2cfa'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:24.125900 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-13b69663-73c6-4076-be3e-5760b4ac9b01 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b2b9097988284575ae365c2d18ae2cfa'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:24.145695 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1368/2734] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:42:23 2022] DELETE /identity/v3/users/b2b9097988284575ae365c2d18ae2cfa => generated 0 bytes in 355 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:42:24.249822 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62be350e-7624-4cd8-8fdc-a3fae4fffed4 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:24.251847 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-62be350e-7624-4cd8-8fdc-a3fae4fffed4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:24.356117 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62be350e-7624-4cd8-8fdc-a3fae4fffed4 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:24.435159 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dff69d6a-c1d7-4d17-b8a6-a98f474949e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:24.436014 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dff69d6a-c1d7-4d17-b8a6-a98f474949e1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:24.436327 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dff69d6a-c1d7-4d17-b8a6-a98f474949e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:24.438057 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dff69d6a-c1d7-4d17-b8a6-a98f474949e1 admin admin] PATH_INFO: `/v3/projects/69d7f124802b450eae69db4d42b62d83` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:24.439490 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dff69d6a-c1d7-4d17-b8a6-a98f474949e1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=69d7f124802b450eae69db4d42b62d83)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:24.441584 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dff69d6a-c1d7-4d17-b8a6-a98f474949e1 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:24.488479 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:42:24.510554 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13b69663-73c6-4076-be3e-5760b4ac9b01 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:24.512970 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-dff69d6a-c1d7-4d17-b8a6-a98f474949e1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '69d7f124802b450eae69db4d42b62d83', 'request_id': 'req-dff69d6a-c1d7-4d17-b8a6-a98f474949e1'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:24.513485 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-13b69663-73c6-4076-be3e-5760b4ac9b01 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:24.519827 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-dff69d6a-c1d7-4d17-b8a6-a98f474949e1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '69d7f124802b450eae69db4d42b62d83', 'request_id': 'req-dff69d6a-c1d7-4d17-b8a6-a98f474949e1'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:24.527685 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-dff69d6a-c1d7-4d17-b8a6-a98f474949e1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '69d7f124802b450eae69db4d42b62d83', 'request_id': 'req-dff69d6a-c1d7-4d17-b8a6-a98f474949e1'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:24.542107 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13b69663-73c6-4076-be3e-5760b4ac9b01 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:24.543348 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-dff69d6a-c1d7-4d17-b8a6-a98f474949e1 admin admin] The token cache is being invalidate because project 69d7f124802b450eae69db4d42b62d83 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81434) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:42:24.543646 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-dff69d6a-c1d7-4d17-b8a6-a98f474949e1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:24.561844 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-dff69d6a-c1d7-4d17-b8a6-a98f474949e1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:24.568817 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1367/2735] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:42:24 2022] DELETE /identity/v3/projects/69d7f124802b450eae69db4d42b62d83 => generated 0 bytes in 321 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:42:24.573011 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dff69d6a-c1d7-4d17-b8a6-a98f474949e1 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:24.574461 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-dff69d6a-c1d7-4d17-b8a6-a98f474949e1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:24.630440 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f2260d9-8d6b-44ad-8e86-ab8d2d88a3f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:24.631188 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f2260d9-8d6b-44ad-8e86-ab8d2d88a3f7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:24.631360 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f2260d9-8d6b-44ad-8e86-ab8d2d88a3f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:24.631830 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f2260d9-8d6b-44ad-8e86-ab8d2d88a3f7 admin admin] PATH_INFO: `/v3/projects/57208a72edb44fc0a76a54748420feca` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:24.633001 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f2260d9-8d6b-44ad-8e86-ab8d2d88a3f7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=57208a72edb44fc0a76a54748420feca)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:24.634965 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f2260d9-8d6b-44ad-8e86-ab8d2d88a3f7 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:24.704814 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dff69d6a-c1d7-4d17-b8a6-a98f474949e1 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:24.715937 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-3f2260d9-8d6b-44ad-8e86-ab8d2d88a3f7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '57208a72edb44fc0a76a54748420feca', 'request_id': 'req-3f2260d9-8d6b-44ad-8e86-ab8d2d88a3f7'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:24.724112 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-3f2260d9-8d6b-44ad-8e86-ab8d2d88a3f7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '57208a72edb44fc0a76a54748420feca', 'request_id': 'req-3f2260d9-8d6b-44ad-8e86-ab8d2d88a3f7'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:24.747755 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-3f2260d9-8d6b-44ad-8e86-ab8d2d88a3f7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '57208a72edb44fc0a76a54748420feca', 'request_id': 'req-3f2260d9-8d6b-44ad-8e86-ab8d2d88a3f7'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:24.767188 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-3f2260d9-8d6b-44ad-8e86-ab8d2d88a3f7 admin admin] The token cache is being invalidate because project 57208a72edb44fc0a76a54748420feca was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:42:24.767578 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-3f2260d9-8d6b-44ad-8e86-ab8d2d88a3f7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:24.768991 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-3f2260d9-8d6b-44ad-8e86-ab8d2d88a3f7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:24.771802 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1369/2736] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:42:24 2022] DELETE /identity/v3/projects/57208a72edb44fc0a76a54748420feca => generated 0 bytes in 262 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:42:24.776562 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4808afbc-47f6-480c-9258-c056ea121cf8 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:24.777058 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4808afbc-47f6-480c-9258-c056ea121cf8 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:24.777538 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4808afbc-47f6-480c-9258-c056ea121cf8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:24.795863 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef10fdf0-939b-48b3-bae3-5593d559a14d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:24.796748 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef10fdf0-939b-48b3-bae3-5593d559a14d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:24.797194 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef10fdf0-939b-48b3-bae3-5593d559a14d admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:24.797671 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef10fdf0-939b-48b3-bae3-5593d559a14d admin admin] PATH_INFO: `/v3/users/5c64b71b38a24a9d8242e8cae089410a` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:24.799959 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef10fdf0-939b-48b3-bae3-5593d559a14d admin admin] RBAC: Authorizing `identity:delete_user(user_id=5c64b71b38a24a9d8242e8cae089410a)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:24.803095 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef10fdf0-939b-48b3-bae3-5593d559a14d admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:24.806272 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-4808afbc-47f6-480c-9258-c056ea121cf8 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:42:24.807510 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-4808afbc-47f6-480c-9258-c056ea121cf8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:24.817948 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1370/2737] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:42:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 21:42:24.825809 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac9efda0-16ee-474a-b8f9-48ad4d7c354e None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:24.826671 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac9efda0-16ee-474a-b8f9-48ad4d7c354e None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:24.827545 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac9efda0-16ee-474a-b8f9-48ad4d7c354e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:24.860345 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-ac9efda0-16ee-474a-b8f9-48ad4d7c354e None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:42:24.861605 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-ac9efda0-16ee-474a-b8f9-48ad4d7c354e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:24.871691 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1371/2738] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:42:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 47 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 21:42:24.876696 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac9efda0-16ee-474a-b8f9-48ad4d7c354e None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:24.879769 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-ef10fdf0-939b-48b3-bae3-5593d559a14d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5c64b71b38a24a9d8242e8cae089410a'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:24.887674 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-ef10fdf0-939b-48b3-bae3-5593d559a14d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5c64b71b38a24a9d8242e8cae089410a'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:24.890538 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac9efda0-16ee-474a-b8f9-48ad4d7c354e None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:24.895683 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e432c601-8dbd-477a-82e8-8eabf8d62854 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:24.896352 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e432c601-8dbd-477a-82e8-8eabf8d62854 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:24.896595 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e432c601-8dbd-477a-82e8-8eabf8d62854 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:24.896867 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e432c601-8dbd-477a-82e8-8eabf8d62854 admin admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:24.897543 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e432c601-8dbd-477a-82e8-8eabf8d62854 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:42:24.897876 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e432c601-8dbd-477a-82e8-8eabf8d62854 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:24.898011 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-ef10fdf0-939b-48b3-bae3-5593d559a14d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5c64b71b38a24a9d8242e8cae089410a'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:24.898664 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-ef10fdf0-939b-48b3-bae3-5593d559a14d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5c64b71b38a24a9d8242e8cae089410a'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:24.899298 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-ef10fdf0-939b-48b3-bae3-5593d559a14d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5c64b71b38a24a9d8242e8cae089410a'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:24.899478 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e432c601-8dbd-477a-82e8-8eabf8d62854 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:24.900853 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:DELETE=14,keystone:SELECT=84,keystone:INSERT=2 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:42:24.914309 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1372/2739] 10.222.0.90 () {58 vars in 1153 bytes} [Mon Aug 29 21:42:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:42:24.918377 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e432c601-8dbd-477a-82e8-8eabf8d62854 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:24.920452 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-e432c601-8dbd-477a-82e8-8eabf8d62854 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:24.928150 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1368/2740] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:42:24 2022] DELETE /identity/v3/users/5c64b71b38a24a9d8242e8cae089410a => generated 0 bytes in 356 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 21:42:24.937416 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e432c601-8dbd-477a-82e8-8eabf8d62854 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:24.953663 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:42:25.035062 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edbf1f84-4a7b-429a-86a7-1881a189ff04 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:25.036225 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edbf1f84-4a7b-429a-86a7-1881a189ff04 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:25.036754 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edbf1f84-4a7b-429a-86a7-1881a189ff04 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:25.037215 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edbf1f84-4a7b-429a-86a7-1881a189ff04 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:25.038315 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edbf1f84-4a7b-429a-86a7-1881a189ff04 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:25.041978 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edbf1f84-4a7b-429a-86a7-1881a189ff04 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:25.088502 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1373/2741] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:42:24 2022] POST /identity/v3/projects => generated 385 bytes in 171 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 21:42:25.092752 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef10fdf0-939b-48b3-bae3-5593d559a14d admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:25.103536 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef10fdf0-939b-48b3-bae3-5593d559a14d admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:25.137229 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edbf1f84-4a7b-429a-86a7-1881a189ff04 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:25.140470 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-edbf1f84-4a7b-429a-86a7-1881a189ff04 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:25.190916 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71d2a29b-1ea0-43ab-a85a-16b82dda1178 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:25.191665 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71d2a29b-1ea0-43ab-a85a-16b82dda1178 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:25.191987 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71d2a29b-1ea0-43ab-a85a-16b82dda1178 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:25.192263 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71d2a29b-1ea0-43ab-a85a-16b82dda1178 admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:25.192922 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71d2a29b-1ea0-43ab-a85a-16b82dda1178 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:25.195272 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71d2a29b-1ea0-43ab-a85a-16b82dda1178 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:25.223794 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1369/2742] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:42:25 2022] POST /identity/v3/users => generated 293 bytes in 131 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:42:25.226271 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edbf1f84-4a7b-429a-86a7-1881a189ff04 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:25.228421 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71d2a29b-1ea0-43ab-a85a-16b82dda1178 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:25.233611 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afb824eb-e7c8-426d-998c-67e398d50e91 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:25.234332 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afb824eb-e7c8-426d-998c-67e398d50e91 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:25.234607 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afb824eb-e7c8-426d-998c-67e398d50e91 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:25.235142 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afb824eb-e7c8-426d-998c-67e398d50e91 admin admin] PATH_INFO: `/v3/projects/c77def5a174a4c45a37f0eea07e62368` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:25.237382 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afb824eb-e7c8-426d-998c-67e398d50e91 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c77def5a174a4c45a37f0eea07e62368)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:25.239525 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afb824eb-e7c8-426d-998c-67e398d50e91 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:25.241500 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71d2a29b-1ea0-43ab-a85a-16b82dda1178 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:25.247108 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d8993c8-f778-4725-af3d-fd39ebbfe214 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:25.248232 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d8993c8-f778-4725-af3d-fd39ebbfe214 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:25.248517 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d8993c8-f778-4725-af3d-fd39ebbfe214 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:25.248795 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d8993c8-f778-4725-af3d-fd39ebbfe214 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:25.250377 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d8993c8-f778-4725-af3d-fd39ebbfe214 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:25.252494 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d8993c8-f778-4725-af3d-fd39ebbfe214 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:25.265828 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1370/2743] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:42:25 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 29 21:42:25.269732 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d8993c8-f778-4725-af3d-fd39ebbfe214 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:25.282642 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d8993c8-f778-4725-af3d-fd39ebbfe214 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:25.288304 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc304d5f-272c-4523-a7bd-378935633e5e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:25.288963 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc304d5f-272c-4523-a7bd-378935633e5e admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:25.289561 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc304d5f-272c-4523-a7bd-378935633e5e admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:25.289561 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc304d5f-272c-4523-a7bd-378935633e5e admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:25.290328 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc304d5f-272c-4523-a7bd-378935633e5e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:25.292104 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc304d5f-272c-4523-a7bd-378935633e5e admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:25.307840 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1371/2744] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:42:25 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:42:25.311322 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-afb824eb-e7c8-426d-998c-67e398d50e91 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c77def5a174a4c45a37f0eea07e62368', 'request_id': 'req-afb824eb-e7c8-426d-998c-67e398d50e91'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:25.313129 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc304d5f-272c-4523-a7bd-378935633e5e admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:25.315916 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-afb824eb-e7c8-426d-998c-67e398d50e91 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c77def5a174a4c45a37f0eea07e62368', 'request_id': 'req-afb824eb-e7c8-426d-998c-67e398d50e91'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:25.322434 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-afb824eb-e7c8-426d-998c-67e398d50e91 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c77def5a174a4c45a37f0eea07e62368', 'request_id': 'req-afb824eb-e7c8-426d-998c-67e398d50e91'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:25.335672 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc304d5f-272c-4523-a7bd-378935633e5e admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:25.338294 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-afb824eb-e7c8-426d-998c-67e398d50e91 admin admin] The token cache is being invalidate because project c77def5a174a4c45a37f0eea07e62368 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:42:25.338682 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-afb824eb-e7c8-426d-998c-67e398d50e91 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:25.339558 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-afb824eb-e7c8-426d-998c-67e398d50e91 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:25.341653 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1374/2745] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:42:25 2022] DELETE /identity/v3/projects/c77def5a174a4c45a37f0eea07e62368 => generated 0 bytes in 205 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 21:42:25.351717 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f8aa52-9464-4383-b646-5a4b8114cac7 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:25.352314 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f8aa52-9464-4383-b646-5a4b8114cac7 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:25.352781 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f8aa52-9464-4383-b646-5a4b8114cac7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:25.377415 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-47f8aa52-9464-4383-b646-5a4b8114cac7 None None] MFA Rules not processed for user `0c0b0da87b124de1be00faec4bbdb404`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:42:25.378839 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-47f8aa52-9464-4383-b646-5a4b8114cac7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:25.413203 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:42:25.418460 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-874d03aa-ac15-409c-93d2-5694f3554678 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:25.420253 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-874d03aa-ac15-409c-93d2-5694f3554678 admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:25.420569 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-874d03aa-ac15-409c-93d2-5694f3554678 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:25.421317 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-874d03aa-ac15-409c-93d2-5694f3554678 admin admin] PATH_INFO: `/v3/projects/0c295e9f2c43473b8ec4448794589ef2/users/2cdcf48a1bea4f71b1428149afac3749/roles/02e242d5520740d4b77a5f4008c7eb57` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:25.442637 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-874d03aa-ac15-409c-93d2-5694f3554678 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0c295e9f2c43473b8ec4448794589ef2, user_id=2cdcf48a1bea4f71b1428149afac3749, role_id=02e242d5520740d4b77a5f4008c7eb57)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:25.445343 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-874d03aa-ac15-409c-93d2-5694f3554678 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:25.449481 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1375/2746] 10.222.0.90 () {58 vars in 967 bytes} [Mon Aug 29 21:42:25 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 21:42:25.462353 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1372/2747] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:42:25 2022] PUT /identity/v3/projects/0c295e9f2c43473b8ec4448794589ef2/users/2cdcf48a1bea4f71b1428149afac3749/roles/02e242d5520740d4b77a5f4008c7eb57 => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:42:25.466375 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47f8aa52-9464-4383-b646-5a4b8114cac7 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:25.468757 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-47f8aa52-9464-4383-b646-5a4b8114cac7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:25.476270 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98dafe0d-944c-4cad-80fe-9c81cee28f56 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:25.476599 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98dafe0d-944c-4cad-80fe-9c81cee28f56 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:25.477109 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98dafe0d-944c-4cad-80fe-9c81cee28f56 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:25.513574 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-98dafe0d-944c-4cad-80fe-9c81cee28f56 None None] MFA Rules not processed for user `2cdcf48a1bea4f71b1428149afac3749`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:42:25.515708 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-98dafe0d-944c-4cad-80fe-9c81cee28f56 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:25.538934 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=94,keystone:DELETE=2,keystone:INSERT=4 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:42:25.556005 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47f8aa52-9464-4383-b646-5a4b8114cac7 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:25.638202 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deb4efd1-cc92-4602-ae15-2b9fc961599c service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '139bbbde6d884b42b5440727926e2c58', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:25.638911 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deb4efd1-cc92-4602-ae15-2b9fc961599c service swift] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:25.639149 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deb4efd1-cc92-4602-ae15-2b9fc961599c service swift] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:25.639936 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deb4efd1-cc92-4602-ae15-2b9fc961599c service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:25.713539 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-deb4efd1-cc92-4602-ae15-2b9fc961599c service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:25.715217 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-deb4efd1-cc92-4602-ae15-2b9fc961599c service swift] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:25.726086 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1376/2748] 10.222.0.90 () {60 vars in 1384 bytes} [Mon Aug 29 21:42:25 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 260 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Aug 29 21:42:25.732433 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1373/2749] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:42:25 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 258 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:42:25.738902 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deb4efd1-cc92-4602-ae15-2b9fc961599c service swift] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:25.740328 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-deb4efd1-cc92-4602-ae15-2b9fc961599c service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:25.775500 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:42:25.825030 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deb4efd1-cc92-4602-ae15-2b9fc961599c service swift] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:25.831732 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5314080c-fcd4-4bb9-8bcf-a3ad4410facd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:25.832597 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5314080c-fcd4-4bb9-8bcf-a3ad4410facd admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:25.833067 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5314080c-fcd4-4bb9-8bcf-a3ad4410facd admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:25.833531 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5314080c-fcd4-4bb9-8bcf-a3ad4410facd admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:25.834687 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5314080c-fcd4-4bb9-8bcf-a3ad4410facd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:25.837670 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5314080c-fcd4-4bb9-8bcf-a3ad4410facd admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:25.876278 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1377/2750] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:42:25 2022] POST /identity/v3/projects => generated 387 bytes in 138 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:42:25.881259 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98dafe0d-944c-4cad-80fe-9c81cee28f56 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:25.896738 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98dafe0d-944c-4cad-80fe-9c81cee28f56 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:25.965359 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbb78625-7fae-4fcb-820c-3b86b41797d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:25.966002 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb78625-7fae-4fcb-820c-3b86b41797d1 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:25.966282 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb78625-7fae-4fcb-820c-3b86b41797d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:25.966581 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb78625-7fae-4fcb-820c-3b86b41797d1 admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:25.967221 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbb78625-7fae-4fcb-820c-3b86b41797d1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:25.969634 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbb78625-7fae-4fcb-820c-3b86b41797d1 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:25.993392 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1374/2751] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:42:25 2022] POST /identity/v3/users => generated 300 bytes in 113 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 21:42:25.997148 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5314080c-fcd4-4bb9-8bcf-a3ad4410facd admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:26.010299 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5314080c-fcd4-4bb9-8bcf-a3ad4410facd admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:26.015926 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c1580f8-b284-4f69-8bf2-b7b7051d875b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:26.016584 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c1580f8-b284-4f69-8bf2-b7b7051d875b admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:26.016865 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c1580f8-b284-4f69-8bf2-b7b7051d875b admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:26.017134 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c1580f8-b284-4f69-8bf2-b7b7051d875b admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:26.017777 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c1580f8-b284-4f69-8bf2-b7b7051d875b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:26.019401 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c1580f8-b284-4f69-8bf2-b7b7051d875b admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:26.037872 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1378/2752] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:42:25 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:42:26.040962 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbb78625-7fae-4fcb-820c-3b86b41797d1 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:26.051613 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbb78625-7fae-4fcb-820c-3b86b41797d1 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:26.057018 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d6cdad0-7b17-48f2-ab60-cccd0cebc225 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:26.057727 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d6cdad0-7b17-48f2-ab60-cccd0cebc225 admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:26.058053 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d6cdad0-7b17-48f2-ab60-cccd0cebc225 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:26.058349 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d6cdad0-7b17-48f2-ab60-cccd0cebc225 admin admin] PATH_INFO: `/v3/projects/6db1047c66a140ba84e070039979497e/users/8e08d01bfc5548999004ac02f746bd30/roles/a6e562cdc96c4be691d638990f8403fd` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:26.081554 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d6cdad0-7b17-48f2-ab60-cccd0cebc225 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6db1047c66a140ba84e070039979497e, user_id=8e08d01bfc5548999004ac02f746bd30, role_id=a6e562cdc96c4be691d638990f8403fd)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:26.084486 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d6cdad0-7b17-48f2-ab60-cccd0cebc225 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:26.103304 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1375/2753] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:42:26 2022] PUT /identity/v3/projects/6db1047c66a140ba84e070039979497e/users/8e08d01bfc5548999004ac02f746bd30/roles/a6e562cdc96c4be691d638990f8403fd => generated 0 bytes in 63 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:42:26.107320 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c1580f8-b284-4f69-8bf2-b7b7051d875b admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:26.119400 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c1580f8-b284-4f69-8bf2-b7b7051d875b admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:26.204927 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff94f1c8-e25a-4965-972a-e56d0386396d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:26.205647 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff94f1c8-e25a-4965-972a-e56d0386396d admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:26.206029 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff94f1c8-e25a-4965-972a-e56d0386396d admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:26.206654 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff94f1c8-e25a-4965-972a-e56d0386396d admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:26.207478 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff94f1c8-e25a-4965-972a-e56d0386396d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:26.210554 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff94f1c8-e25a-4965-972a-e56d0386396d admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:26.226885 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1379/2754] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:42:26 2022] GET /identity/v3/roles => generated 1420 bytes in 120 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 29 21:42:26.231625 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d6cdad0-7b17-48f2-ab60-cccd0cebc225 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:26.244585 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d6cdad0-7b17-48f2-ab60-cccd0cebc225 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:26.249808 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2cca801-176e-4885-9291-27a4bbf45d68 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:26.250680 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2cca801-176e-4885-9291-27a4bbf45d68 admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:26.251319 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2cca801-176e-4885-9291-27a4bbf45d68 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:26.251448 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2cca801-176e-4885-9291-27a4bbf45d68 admin admin] PATH_INFO: `/v3/projects/6db1047c66a140ba84e070039979497e/users/8e08d01bfc5548999004ac02f746bd30/roles/a6e562cdc96c4be691d638990f8403fd` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:26.253347 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2cca801-176e-4885-9291-27a4bbf45d68 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6db1047c66a140ba84e070039979497e, user_id=8e08d01bfc5548999004ac02f746bd30, role_id=a6e562cdc96c4be691d638990f8403fd)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:26.256149 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2cca801-176e-4885-9291-27a4bbf45d68 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:26.267282 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1376/2755] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:42:26 2022] PUT /identity/v3/projects/6db1047c66a140ba84e070039979497e/users/8e08d01bfc5548999004ac02f746bd30/roles/a6e562cdc96c4be691d638990f8403fd => generated 0 bytes in 36 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:42:26.277550 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72869723-014f-4229-96ca-500dd9900279 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:26.277859 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72869723-014f-4229-96ca-500dd9900279 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:26.278136 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72869723-014f-4229-96ca-500dd9900279 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:26.310823 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-72869723-014f-4229-96ca-500dd9900279 None None] MFA Rules not processed for user `8e08d01bfc5548999004ac02f746bd30`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:42:26.312819 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-72869723-014f-4229-96ca-500dd9900279 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:26.396868 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1380/2756] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:42:26 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:42:26.409608 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2cca801-176e-4885-9291-27a4bbf45d68 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:26.411828 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-c2cca801-176e-4885-9291-27a4bbf45d68 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:26.482596 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2cca801-176e-4885-9291-27a4bbf45d68 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:26.486263 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08626d73-2b24-4020-ab3e-81725d7ae1d2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dc226f7ddcb49ecacb51ce57fdfc11b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:26.486863 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08626d73-2b24-4020-ab3e-81725d7ae1d2 service cinder] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:26.487137 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08626d73-2b24-4020-ab3e-81725d7ae1d2 service cinder] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:26.487406 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08626d73-2b24-4020-ab3e-81725d7ae1d2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:26.496570 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08626d73-2b24-4020-ab3e-81725d7ae1d2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:26.498024 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08626d73-2b24-4020-ab3e-81725d7ae1d2 service cinder] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:26.508038 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1377/2757] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:42:26 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:42:26.539204 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72869723-014f-4229-96ca-500dd9900279 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:26.559245 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:42:26.605048 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72869723-014f-4229-96ca-500dd9900279 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:26.611180 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da944ce3-c693-49cf-9268-8f46b6994cd2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:26.611721 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da944ce3-c693-49cf-9268-8f46b6994cd2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:26.612178 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da944ce3-c693-49cf-9268-8f46b6994cd2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:26.612454 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da944ce3-c693-49cf-9268-8f46b6994cd2 admin admin] PATH_INFO: `/v3/users/2cdcf48a1bea4f71b1428149afac3749` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:26.614093 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da944ce3-c693-49cf-9268-8f46b6994cd2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2cdcf48a1bea4f71b1428149afac3749)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:26.616343 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da944ce3-c693-49cf-9268-8f46b6994cd2 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:26.668238 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-da944ce3-c693-49cf-9268-8f46b6994cd2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2cdcf48a1bea4f71b1428149afac3749'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:26.680281 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-da944ce3-c693-49cf-9268-8f46b6994cd2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2cdcf48a1bea4f71b1428149afac3749'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:26.680961 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-da944ce3-c693-49cf-9268-8f46b6994cd2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2cdcf48a1bea4f71b1428149afac3749'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:26.681569 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-da944ce3-c693-49cf-9268-8f46b6994cd2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2cdcf48a1bea4f71b1428149afac3749'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:26.688889 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-da944ce3-c693-49cf-9268-8f46b6994cd2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2cdcf48a1bea4f71b1428149afac3749'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:26.702851 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1381/2758] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:42:26 2022] DELETE /identity/v3/users/2cdcf48a1bea4f71b1428149afac3749 => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:42:26.707456 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec56415c-f283-4eca-af2f-7580fefb926f None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:26.707747 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec56415c-f283-4eca-af2f-7580fefb926f None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:26.708028 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec56415c-f283-4eca-af2f-7580fefb926f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:26.727351 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-ec56415c-f283-4eca-af2f-7580fefb926f None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:42:26.728586 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-ec56415c-f283-4eca-af2f-7580fefb926f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:26.738542 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:42:26.791127 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1378/2759] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:42:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 21:42:26.801994 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da944ce3-c693-49cf-9268-8f46b6994cd2 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:26.803608 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-da944ce3-c693-49cf-9268-8f46b6994cd2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:26.870397 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da944ce3-c693-49cf-9268-8f46b6994cd2 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:26.877054 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4caaec3-67ea-43b8-91fa-ab71d33f2d7f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:26.877665 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4caaec3-67ea-43b8-91fa-ab71d33f2d7f service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:26.877985 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4caaec3-67ea-43b8-91fa-ab71d33f2d7f service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:26.878262 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4caaec3-67ea-43b8-91fa-ab71d33f2d7f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:26.888270 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4caaec3-67ea-43b8-91fa-ab71d33f2d7f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:26.889691 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4caaec3-67ea-43b8-91fa-ab71d33f2d7f service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:26.899186 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1382/2760] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:42:26 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:42:27.176110 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec56415c-f283-4eca-af2f-7580fefb926f None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:27.177710 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-ec56415c-f283-4eca-af2f-7580fefb926f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:27.192396 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec56415c-f283-4eca-af2f-7580fefb926f None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:27.197372 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-659d1a7d-a7c9-4e7a-912e-c9c106644e45 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:27.198165 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-659d1a7d-a7c9-4e7a-912e-c9c106644e45 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:27.198544 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-659d1a7d-a7c9-4e7a-912e-c9c106644e45 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:27.200321 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-659d1a7d-a7c9-4e7a-912e-c9c106644e45 admin admin] PATH_INFO: `/v3/projects/0c295e9f2c43473b8ec4448794589ef2` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:27.201287 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-659d1a7d-a7c9-4e7a-912e-c9c106644e45 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0c295e9f2c43473b8ec4448794589ef2)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:27.203466 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-659d1a7d-a7c9-4e7a-912e-c9c106644e45 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:27.277065 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-659d1a7d-a7c9-4e7a-912e-c9c106644e45 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0c295e9f2c43473b8ec4448794589ef2', 'request_id': 'req-659d1a7d-a7c9-4e7a-912e-c9c106644e45'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:27.284912 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-659d1a7d-a7c9-4e7a-912e-c9c106644e45 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0c295e9f2c43473b8ec4448794589ef2', 'request_id': 'req-659d1a7d-a7c9-4e7a-912e-c9c106644e45'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:27.291543 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-659d1a7d-a7c9-4e7a-912e-c9c106644e45 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0c295e9f2c43473b8ec4448794589ef2', 'request_id': 'req-659d1a7d-a7c9-4e7a-912e-c9c106644e45'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:27.306823 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-659d1a7d-a7c9-4e7a-912e-c9c106644e45 admin admin] The token cache is being invalidate because project 0c295e9f2c43473b8ec4448794589ef2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81434) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:42:27.307119 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-659d1a7d-a7c9-4e7a-912e-c9c106644e45 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:27.307757 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-659d1a7d-a7c9-4e7a-912e-c9c106644e45 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:27.309384 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1379/2761] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:42:27 2022] DELETE /identity/v3/projects/0c295e9f2c43473b8ec4448794589ef2 => generated 0 bytes in 134 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:42:27.313510 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4caaec3-67ea-43b8-91fa-ab71d33f2d7f service neutron] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:27.323943 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4caaec3-67ea-43b8-91fa-ab71d33f2d7f service neutron] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:27.382698 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:42:27.416839 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5297e707-cb4e-448f-b4d7-e152816b1ef9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:27.417697 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5297e707-cb4e-448f-b4d7-e152816b1ef9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:27.418060 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5297e707-cb4e-448f-b4d7-e152816b1ef9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:27.418449 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5297e707-cb4e-448f-b4d7-e152816b1ef9 admin admin] PATH_INFO: `/v3/users/8e08d01bfc5548999004ac02f746bd30` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:27.420164 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5297e707-cb4e-448f-b4d7-e152816b1ef9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8e08d01bfc5548999004ac02f746bd30)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:27.423596 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5297e707-cb4e-448f-b4d7-e152816b1ef9 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:27.483964 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-5297e707-cb4e-448f-b4d7-e152816b1ef9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8e08d01bfc5548999004ac02f746bd30'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:27.505721 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-5297e707-cb4e-448f-b4d7-e152816b1ef9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8e08d01bfc5548999004ac02f746bd30'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:27.506517 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-5297e707-cb4e-448f-b4d7-e152816b1ef9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8e08d01bfc5548999004ac02f746bd30'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:27.507140 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-5297e707-cb4e-448f-b4d7-e152816b1ef9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8e08d01bfc5548999004ac02f746bd30'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:27.518075 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-5297e707-cb4e-448f-b4d7-e152816b1ef9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8e08d01bfc5548999004ac02f746bd30'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:27.535969 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1383/2762] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:42:27 2022] DELETE /identity/v3/users/8e08d01bfc5548999004ac02f746bd30 => generated 0 bytes in 223 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:42:27.831872 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-659d1a7d-a7c9-4e7a-912e-c9c106644e45 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:27.834586 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-659d1a7d-a7c9-4e7a-912e-c9c106644e45 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:27.850082 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-659d1a7d-a7c9-4e7a-912e-c9c106644e45 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:27.863675 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5297e707-cb4e-448f-b4d7-e152816b1ef9 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:27.865190 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-5297e707-cb4e-448f-b4d7-e152816b1ef9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:27.928452 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f78a0b4-bc2e-478e-a73f-8d108438f5de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:27.929127 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f78a0b4-bc2e-478e-a73f-8d108438f5de admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:27.929410 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f78a0b4-bc2e-478e-a73f-8d108438f5de admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:27.929680 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f78a0b4-bc2e-478e-a73f-8d108438f5de admin admin] PATH_INFO: `/v3/users/7952b085a8b94dcbbeb6a2817ad602e9` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:27.930832 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f78a0b4-bc2e-478e-a73f-8d108438f5de admin admin] RBAC: Authorizing `identity:delete_user(user_id=7952b085a8b94dcbbeb6a2817ad602e9)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:27.932762 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f78a0b4-bc2e-478e-a73f-8d108438f5de admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:27.939895 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5297e707-cb4e-448f-b4d7-e152816b1ef9 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:27.944517 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbbc01a7-923a-4d7b-b726-1e4b8edc1916 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:27.945155 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbbc01a7-923a-4d7b-b726-1e4b8edc1916 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:27.945436 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbbc01a7-923a-4d7b-b726-1e4b8edc1916 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:27.945713 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbbc01a7-923a-4d7b-b726-1e4b8edc1916 admin admin] PATH_INFO: `/v3/projects/6db1047c66a140ba84e070039979497e` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:27.946881 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbbc01a7-923a-4d7b-b726-1e4b8edc1916 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6db1047c66a140ba84e070039979497e)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:27.948788 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbbc01a7-923a-4d7b-b726-1e4b8edc1916 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:27.980326 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:42:27.987355 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-1f78a0b4-bc2e-478e-a73f-8d108438f5de admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7952b085a8b94dcbbeb6a2817ad602e9'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:27.995844 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-1f78a0b4-bc2e-478e-a73f-8d108438f5de admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7952b085a8b94dcbbeb6a2817ad602e9'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:28.002053 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-fbbc01a7-923a-4d7b-b726-1e4b8edc1916 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6db1047c66a140ba84e070039979497e', 'request_id': 'req-fbbc01a7-923a-4d7b-b726-1e4b8edc1916'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:28.004393 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-1f78a0b4-bc2e-478e-a73f-8d108438f5de admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7952b085a8b94dcbbeb6a2817ad602e9'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:28.005163 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-1f78a0b4-bc2e-478e-a73f-8d108438f5de admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7952b085a8b94dcbbeb6a2817ad602e9'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:28.005886 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-1f78a0b4-bc2e-478e-a73f-8d108438f5de admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7952b085a8b94dcbbeb6a2817ad602e9'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:28.010152 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-fbbc01a7-923a-4d7b-b726-1e4b8edc1916 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6db1047c66a140ba84e070039979497e', 'request_id': 'req-fbbc01a7-923a-4d7b-b726-1e4b8edc1916'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:28.016467 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=83,keystone:DELETE=16,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:42:28.018330 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-fbbc01a7-923a-4d7b-b726-1e4b8edc1916 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6db1047c66a140ba84e070039979497e', 'request_id': 'req-fbbc01a7-923a-4d7b-b726-1e4b8edc1916'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:28.034171 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1380/2763] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:42:27 2022] DELETE /identity/v3/users/7952b085a8b94dcbbeb6a2817ad602e9 => generated 0 bytes in 204 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:42:28.036048 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-fbbc01a7-923a-4d7b-b726-1e4b8edc1916 admin admin] The token cache is being invalidate because project 6db1047c66a140ba84e070039979497e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:42:28.036337 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-fbbc01a7-923a-4d7b-b726-1e4b8edc1916 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:28.037000 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-fbbc01a7-923a-4d7b-b726-1e4b8edc1916 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:28.038708 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1384/2764] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:42:27 2022] DELETE /identity/v3/projects/6db1047c66a140ba84e070039979497e => generated 0 bytes in 175 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 21:42:28.043119 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62a0e325-5fd9-4858-ba10-a122738b5e56 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:28.043473 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62a0e325-5fd9-4858-ba10-a122738b5e56 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:28.043820 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62a0e325-5fd9-4858-ba10-a122738b5e56 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:28.064962 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-62a0e325-5fd9-4858-ba10-a122738b5e56 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:42:28.066206 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-62a0e325-5fd9-4858-ba10-a122738b5e56 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:28.130704 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1381/2765] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:42:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 21:42:28.138564 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec858c6-8649-45ce-8bb1-13ec6465cacb None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:28.138930 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec858c6-8649-45ce-8bb1-13ec6465cacb None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:28.139936 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec858c6-8649-45ce-8bb1-13ec6465cacb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:28.161503 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-8ec858c6-8649-45ce-8bb1-13ec6465cacb None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:42:28.162825 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-8ec858c6-8649-45ce-8bb1-13ec6465cacb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:28.171741 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1385/2766] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:42:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:42:28.176204 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62a0e325-5fd9-4858-ba10-a122738b5e56 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:28.185719 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62a0e325-5fd9-4858-ba10-a122738b5e56 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:28.191368 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb8fc53c-9c0f-4b05-ad7b-5e75c9aec6d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:28.192037 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb8fc53c-9c0f-4b05-ad7b-5e75c9aec6d8 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:28.192321 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb8fc53c-9c0f-4b05-ad7b-5e75c9aec6d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:28.192623 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb8fc53c-9c0f-4b05-ad7b-5e75c9aec6d8 admin admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:28.193287 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb8fc53c-9c0f-4b05-ad7b-5e75c9aec6d8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:42:28.193589 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb8fc53c-9c0f-4b05-ad7b-5e75c9aec6d8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:28.195213 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb8fc53c-9c0f-4b05-ad7b-5e75c9aec6d8 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:28.208326 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1382/2767] 10.222.0.90 () {58 vars in 1153 bytes} [Mon Aug 29 21:42:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:42:28.213751 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ec858c6-8649-45ce-8bb1-13ec6465cacb None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:28.224097 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb8fc53c-9c0f-4b05-ad7b-5e75c9aec6d8 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:28.224831 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ec858c6-8649-45ce-8bb1-13ec6465cacb None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:28.225908 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-cb8fc53c-9c0f-4b05-ad7b-5e75c9aec6d8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:28.230659 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-093756de-d7b3-4540-b6e1-1b7ccfd72195 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:28.231382 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-093756de-d7b3-4540-b6e1-1b7ccfd72195 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:28.231658 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-093756de-d7b3-4540-b6e1-1b7ccfd72195 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:28.231950 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-093756de-d7b3-4540-b6e1-1b7ccfd72195 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:28.232561 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-093756de-d7b3-4540-b6e1-1b7ccfd72195 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:28.234766 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-093756de-d7b3-4540-b6e1-1b7ccfd72195 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:28.245637 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb8fc53c-9c0f-4b05-ad7b-5e75c9aec6d8 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:28.251017 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d6db786-8228-4d01-8dd8-d455a3a726a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:28.251696 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d6db786-8228-4d01-8dd8-d455a3a726a4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:28.251974 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d6db786-8228-4d01-8dd8-d455a3a726a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:28.252174 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d6db786-8228-4d01-8dd8-d455a3a726a4 admin admin] PATH_INFO: `/v3/projects/902c413d242b4ff7a556c7e487ef5e90` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:28.253351 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d6db786-8228-4d01-8dd8-d455a3a726a4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=902c413d242b4ff7a556c7e487ef5e90)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:28.255695 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d6db786-8228-4d01-8dd8-d455a3a726a4 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:28.275190 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1386/2768] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:42:28 2022] POST /identity/v3/projects => generated 385 bytes in 59 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:42:28.276288 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-093756de-d7b3-4540-b6e1-1b7ccfd72195 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:28.310169 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-4d6db786-8228-4d01-8dd8-d455a3a726a4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '902c413d242b4ff7a556c7e487ef5e90', 'request_id': 'req-4d6db786-8228-4d01-8dd8-d455a3a726a4'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:28.317296 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-4d6db786-8228-4d01-8dd8-d455a3a726a4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '902c413d242b4ff7a556c7e487ef5e90', 'request_id': 'req-4d6db786-8228-4d01-8dd8-d455a3a726a4'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:28.323805 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-4d6db786-8228-4d01-8dd8-d455a3a726a4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '902c413d242b4ff7a556c7e487ef5e90', 'request_id': 'req-4d6db786-8228-4d01-8dd8-d455a3a726a4'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:28.337744 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-4d6db786-8228-4d01-8dd8-d455a3a726a4 admin admin] The token cache is being invalidate because project 902c413d242b4ff7a556c7e487ef5e90 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81434) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:42:28.338080 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-4d6db786-8228-4d01-8dd8-d455a3a726a4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:28.338684 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-4d6db786-8228-4d01-8dd8-d455a3a726a4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:28.340173 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1383/2769] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:42:28 2022] DELETE /identity/v3/projects/902c413d242b4ff7a556c7e487ef5e90 => generated 0 bytes in 117 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:42:28.342053 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-093756de-d7b3-4540-b6e1-1b7ccfd72195 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:28.344974 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a4b66b-7eab-4a34-8709-c9d3df82e009 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:28.345582 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a4b66b-7eab-4a34-8709-c9d3df82e009 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:28.345582 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a4b66b-7eab-4a34-8709-c9d3df82e009 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:28.366133 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-87a4b66b-7eab-4a34-8709-c9d3df82e009 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:42:28.367424 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-87a4b66b-7eab-4a34-8709-c9d3df82e009 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:28.392693 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=87,keystone:DELETE=11,keystone:INSERT=2 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:42:28.425698 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5fbd279-0d50-42f6-ae6f-33a257853f5d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:28.425698 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5fbd279-0d50-42f6-ae6f-33a257853f5d admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:28.425999 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5fbd279-0d50-42f6-ae6f-33a257853f5d admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:28.426250 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5fbd279-0d50-42f6-ae6f-33a257853f5d admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:28.426814 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5fbd279-0d50-42f6-ae6f-33a257853f5d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:28.428777 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5fbd279-0d50-42f6-ae6f-33a257853f5d admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:28.446888 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1384/2770] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:42:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 21:42:28.447914 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87a4b66b-7eab-4a34-8709-c9d3df82e009 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:28.449219 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-87a4b66b-7eab-4a34-8709-c9d3df82e009 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:28.457217 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1387/2771] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:42:28 2022] POST /identity/v3/users => generated 293 bytes in 181 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 21:42:28.458775 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f245b09-f478-4c13-b05e-731a8a385ad4 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:28.459055 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f245b09-f478-4c13-b05e-731a8a385ad4 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:28.459332 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f245b09-f478-4c13-b05e-731a8a385ad4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:28.464360 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87a4b66b-7eab-4a34-8709-c9d3df82e009 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:28.470380 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90777dc7-48cb-4d54-9a74-15ae44e59040 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:28.471237 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90777dc7-48cb-4d54-9a74-15ae44e59040 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:28.471526 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90777dc7-48cb-4d54-9a74-15ae44e59040 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:28.471802 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90777dc7-48cb-4d54-9a74-15ae44e59040 admin admin] PATH_INFO: `/v3/users/862248249d7b408eb5cd18d73733a32c` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:28.473044 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90777dc7-48cb-4d54-9a74-15ae44e59040 admin admin] RBAC: Authorizing `identity:delete_user(user_id=862248249d7b408eb5cd18d73733a32c)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:28.475017 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90777dc7-48cb-4d54-9a74-15ae44e59040 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:28.480057 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-3f245b09-f478-4c13-b05e-731a8a385ad4 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:42:28.481286 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-3f245b09-f478-4c13-b05e-731a8a385ad4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:28.490936 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1388/2772] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:42:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:42:28.491728 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f245b09-f478-4c13-b05e-731a8a385ad4 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:28.492961 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-3f245b09-f478-4c13-b05e-731a8a385ad4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:28.504785 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f245b09-f478-4c13-b05e-731a8a385ad4 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:28.510443 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c05bb25-d262-4006-93ad-8cbd826809a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:28.511129 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c05bb25-d262-4006-93ad-8cbd826809a7 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:28.511476 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c05bb25-d262-4006-93ad-8cbd826809a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:28.511787 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c05bb25-d262-4006-93ad-8cbd826809a7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:28.512435 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c05bb25-d262-4006-93ad-8cbd826809a7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:28.514084 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c05bb25-d262-4006-93ad-8cbd826809a7 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:28.526184 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1389/2773] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:42:28 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:42:28.527020 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c05bb25-d262-4006-93ad-8cbd826809a7 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:28.534511 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-90777dc7-48cb-4d54-9a74-15ae44e59040 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '862248249d7b408eb5cd18d73733a32c'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:28.538553 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c05bb25-d262-4006-93ad-8cbd826809a7 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:28.541490 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-90777dc7-48cb-4d54-9a74-15ae44e59040 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '862248249d7b408eb5cd18d73733a32c'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:28.543679 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba8d6215-5074-4478-b334-fbbc93f8a1b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:28.544428 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba8d6215-5074-4478-b334-fbbc93f8a1b7 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:28.544709 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba8d6215-5074-4478-b334-fbbc93f8a1b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:28.544978 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba8d6215-5074-4478-b334-fbbc93f8a1b7 admin admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:28.546008 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba8d6215-5074-4478-b334-fbbc93f8a1b7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:42:28.546336 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba8d6215-5074-4478-b334-fbbc93f8a1b7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:28.548992 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba8d6215-5074-4478-b334-fbbc93f8a1b7 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:28.553029 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-90777dc7-48cb-4d54-9a74-15ae44e59040 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '862248249d7b408eb5cd18d73733a32c'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:28.553678 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-90777dc7-48cb-4d54-9a74-15ae44e59040 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '862248249d7b408eb5cd18d73733a32c'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:28.554387 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-90777dc7-48cb-4d54-9a74-15ae44e59040 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '862248249d7b408eb5cd18d73733a32c'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:28.560727 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1390/2774] 10.222.0.90 () {58 vars in 1153 bytes} [Mon Aug 29 21:42:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:42:28.561602 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba8d6215-5074-4478-b334-fbbc93f8a1b7 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:28.565708 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-ba8d6215-5074-4478-b334-fbbc93f8a1b7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:28.574073 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1385/2775] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:42:28 2022] DELETE /identity/v3/users/862248249d7b408eb5cd18d73733a32c => generated 0 bytes in 127 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:42:28.574876 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90777dc7-48cb-4d54-9a74-15ae44e59040 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:28.576454 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-90777dc7-48cb-4d54-9a74-15ae44e59040 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:28.580640 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba8d6215-5074-4478-b334-fbbc93f8a1b7 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:28.608493 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=94,keystone:INSERT=5,keystone:DELETE=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:42:28.647257 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee541f15-2490-453e-a5f6-7ddd6a583996 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:28.648152 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee541f15-2490-453e-a5f6-7ddd6a583996 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:28.648596 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee541f15-2490-453e-a5f6-7ddd6a583996 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:28.648983 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee541f15-2490-453e-a5f6-7ddd6a583996 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:28.650097 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee541f15-2490-453e-a5f6-7ddd6a583996 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:28.652106 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee541f15-2490-453e-a5f6-7ddd6a583996 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:28.653345 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90777dc7-48cb-4d54-9a74-15ae44e59040 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:28.658943 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e00ce49a-a574-4563-b298-c1b8be78f11b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:28.659589 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e00ce49a-a574-4563-b298-c1b8be78f11b admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:28.659873 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e00ce49a-a574-4563-b298-c1b8be78f11b admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:28.660156 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e00ce49a-a574-4563-b298-c1b8be78f11b admin admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:28.660799 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e00ce49a-a574-4563-b298-c1b8be78f11b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:28.662736 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e00ce49a-a574-4563-b298-c1b8be78f11b admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:28.664562 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1391/2776] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:42:28 2022] GET /identity/v3/roles => generated 1420 bytes in 103 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:42:28.666080 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38a11f53-f471-472d-9660-9a4284c34bcf None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:28.666461 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38a11f53-f471-472d-9660-9a4284c34bcf None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:28.666824 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38a11f53-f471-472d-9660-9a4284c34bcf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:28.687536 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-38a11f53-f471-472d-9660-9a4284c34bcf None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:42:28.688903 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-38a11f53-f471-472d-9660-9a4284c34bcf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:28.690239 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1386/2777] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:42:28 2022] POST /identity/v3/projects => generated 369 bytes in 116 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:42:28.691058 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e00ce49a-a574-4563-b298-c1b8be78f11b admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:28.732879 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:42:28.772346 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1392/2778] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:42:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:42:28.773202 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38a11f53-f471-472d-9660-9a4284c34bcf None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:28.779702 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e00ce49a-a574-4563-b298-c1b8be78f11b admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:28.786891 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38a11f53-f471-472d-9660-9a4284c34bcf None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:28.787802 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4cecddb-1187-4afe-934c-1c42caaa2bc3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:28.788493 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4cecddb-1187-4afe-934c-1c42caaa2bc3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:28.788775 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4cecddb-1187-4afe-934c-1c42caaa2bc3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:28.789048 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4cecddb-1187-4afe-934c-1c42caaa2bc3 admin admin] PATH_INFO: `/v3/projects/d360668f09044c4bb9c74b45e837e4e0/users/b4a9327b5706420881f989160b1c755b/roles/02e242d5520740d4b77a5f4008c7eb57` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:28.793390 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52fb267b-4317-40c9-92af-8eb8de5befd7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:28.794074 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52fb267b-4317-40c9-92af-8eb8de5befd7 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:28.794354 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52fb267b-4317-40c9-92af-8eb8de5befd7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:28.794625 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52fb267b-4317-40c9-92af-8eb8de5befd7 admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:28.795253 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52fb267b-4317-40c9-92af-8eb8de5befd7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:28.797260 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52fb267b-4317-40c9-92af-8eb8de5befd7 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:28.805453 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4cecddb-1187-4afe-934c-1c42caaa2bc3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d360668f09044c4bb9c74b45e837e4e0, user_id=b4a9327b5706420881f989160b1c755b, role_id=02e242d5520740d4b77a5f4008c7eb57)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:28.812162 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4cecddb-1187-4afe-934c-1c42caaa2bc3 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:28.825147 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1393/2779] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:42:28 2022] POST /identity/v3/users => generated 285 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:42:28.826229 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52fb267b-4317-40c9-92af-8eb8de5befd7 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:28.826607 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1387/2780] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:42:28 2022] PUT /identity/v3/projects/d360668f09044c4bb9c74b45e837e4e0/users/b4a9327b5706420881f989160b1c755b/roles/02e242d5520740d4b77a5f4008c7eb57 => generated 0 bytes in 136 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:42:28.827956 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4cecddb-1187-4afe-934c-1c42caaa2bc3 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:28.829084 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-52fb267b-4317-40c9-92af-8eb8de5befd7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:28.894956 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52fb267b-4317-40c9-92af-8eb8de5befd7 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:28.907246 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4cecddb-1187-4afe-934c-1c42caaa2bc3 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:28.912479 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-950530c9-ba82-486f-836c-819fbf3f416b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:28.913089 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-950530c9-ba82-486f-836c-819fbf3f416b admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:28.913372 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-950530c9-ba82-486f-836c-819fbf3f416b admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:28.913642 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-950530c9-ba82-486f-836c-819fbf3f416b admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:28.914328 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-950530c9-ba82-486f-836c-819fbf3f416b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:28.915938 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-950530c9-ba82-486f-836c-819fbf3f416b admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:28.927630 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1388/2781] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:42:28 2022] GET /identity/v3/roles => generated 1420 bytes in 100 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:42:28.928894 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40c2458d-99d1-4fa1-95b4-84028b0a405a None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:28.929177 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40c2458d-99d1-4fa1-95b4-84028b0a405a None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:28.929455 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40c2458d-99d1-4fa1-95b4-84028b0a405a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:28.950694 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0ad6191-2759-401d-974b-a91d6e978d0f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:28.951567 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0ad6191-2759-401d-974b-a91d6e978d0f service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:28.952607 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0ad6191-2759-401d-974b-a91d6e978d0f service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:28.952874 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0ad6191-2759-401d-974b-a91d6e978d0f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:28.959161 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-40c2458d-99d1-4fa1-95b4-84028b0a405a None None] MFA Rules not processed for user `b4a9327b5706420881f989160b1c755b`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:42:28.960724 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-40c2458d-99d1-4fa1-95b4-84028b0a405a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:28.963504 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0ad6191-2759-401d-974b-a91d6e978d0f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:28.964943 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0ad6191-2759-401d-974b-a91d6e978d0f service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:28.975278 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1394/2782] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:42:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 150 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:42:28.976103 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0ad6191-2759-401d-974b-a91d6e978d0f service neutron] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:28.982294 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:42:28.988300 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0ad6191-2759-401d-974b-a91d6e978d0f service neutron] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:28.996857 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77eb1097-63a1-4dc4-900f-8638aeed8692 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:28.997747 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77eb1097-63a1-4dc4-900f-8638aeed8692 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:28.999455 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77eb1097-63a1-4dc4-900f-8638aeed8692 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:28.999680 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77eb1097-63a1-4dc4-900f-8638aeed8692 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:29.000833 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77eb1097-63a1-4dc4-900f-8638aeed8692 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:29.003912 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77eb1097-63a1-4dc4-900f-8638aeed8692 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:29.029095 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1395/2783] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:42:28 2022] GET /identity/v3/roles => generated 1420 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:42:29.033939 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77eb1097-63a1-4dc4-900f-8638aeed8692 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:29.044791 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77eb1097-63a1-4dc4-900f-8638aeed8692 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:29.051401 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c03b1a9-5518-406c-8dc4-6dad06a5a2f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:29.051947 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c03b1a9-5518-406c-8dc4-6dad06a5a2f8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:29.052245 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c03b1a9-5518-406c-8dc4-6dad06a5a2f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:29.052512 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c03b1a9-5518-406c-8dc4-6dad06a5a2f8 admin admin] PATH_INFO: `/v3/projects/f3ae1ee970c7429a9b68402502e57919/users/f1c6f94c93dd456ca2108516d67032ca/roles/02e242d5520740d4b77a5f4008c7eb57` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:29.062788 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1389/2784] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:42:28 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:42:29.064888 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40c2458d-99d1-4fa1-95b4-84028b0a405a None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:29.071719 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c03b1a9-5518-406c-8dc4-6dad06a5a2f8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f3ae1ee970c7429a9b68402502e57919, user_id=f1c6f94c93dd456ca2108516d67032ca, role_id=02e242d5520740d4b77a5f4008c7eb57)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:29.074643 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c03b1a9-5518-406c-8dc4-6dad06a5a2f8 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:29.077441 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40c2458d-99d1-4fa1-95b4-84028b0a405a None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:29.083479 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dfb369d-24d3-47e7-a0d9-62ca27f6a4dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:29.084262 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dfb369d-24d3-47e7-a0d9-62ca27f6a4dc admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:29.084556 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dfb369d-24d3-47e7-a0d9-62ca27f6a4dc admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:29.085006 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dfb369d-24d3-47e7-a0d9-62ca27f6a4dc admin admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:29.086114 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dfb369d-24d3-47e7-a0d9-62ca27f6a4dc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:29.088446 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dfb369d-24d3-47e7-a0d9-62ca27f6a4dc admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:29.090280 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1396/2785] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:42:29 2022] PUT /identity/v3/projects/f3ae1ee970c7429a9b68402502e57919/users/f1c6f94c93dd456ca2108516d67032ca/roles/02e242d5520740d4b77a5f4008c7eb57 => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 21:42:29.101626 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ffb6f4-3bb2-4717-a9af-586f02d2547f None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:29.101948 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ffb6f4-3bb2-4717-a9af-586f02d2547f None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:29.102226 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ffb6f4-3bb2-4717-a9af-586f02d2547f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:29.122799 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1390/2786] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:42:29 2022] POST /identity/v3/projects => generated 385 bytes in 59 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:42:29.127511 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dfb369d-24d3-47e7-a0d9-62ca27f6a4dc admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:29.143076 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:42:29.147778 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-65ffb6f4-3bb2-4717-a9af-586f02d2547f None None] MFA Rules not processed for user `f1c6f94c93dd456ca2108516d67032ca`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:42:29.151202 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-65ffb6f4-3bb2-4717-a9af-586f02d2547f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:29.206504 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dfb369d-24d3-47e7-a0d9-62ca27f6a4dc admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:29.212695 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4e2d224-09f3-4fb1-985f-6d850c8470ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:29.213406 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4e2d224-09f3-4fb1-985f-6d850c8470ba admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:29.213595 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4e2d224-09f3-4fb1-985f-6d850c8470ba admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:29.213880 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4e2d224-09f3-4fb1-985f-6d850c8470ba admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:29.214519 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4e2d224-09f3-4fb1-985f-6d850c8470ba admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:29.216416 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4e2d224-09f3-4fb1-985f-6d850c8470ba admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:29.241850 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1397/2787] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:42:29 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 21:42:29.245830 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1391/2788] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:42:29 2022] POST /identity/v3/users => generated 299 bytes in 117 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 21:42:29.247457 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65ffb6f4-3bb2-4717-a9af-586f02d2547f None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:29.257020 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4e2d224-09f3-4fb1-985f-6d850c8470ba admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:29.259505 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65ffb6f4-3bb2-4717-a9af-586f02d2547f None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:29.263224 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-b4e2d224-09f3-4fb1-985f-6d850c8470ba admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:29.264811 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff75c2ec-85b8-45ac-8fa6-459f762235c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:29.265425 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff75c2ec-85b8-45ac-8fa6-459f762235c3 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:29.265738 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff75c2ec-85b8-45ac-8fa6-459f762235c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:29.266142 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff75c2ec-85b8-45ac-8fa6-459f762235c3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:29.266843 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff75c2ec-85b8-45ac-8fa6-459f762235c3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:29.268436 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff75c2ec-85b8-45ac-8fa6-459f762235c3 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:29.280309 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1398/2789] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:42:29 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 29 21:42:29.285494 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff75c2ec-85b8-45ac-8fa6-459f762235c3 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:29.298205 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff75c2ec-85b8-45ac-8fa6-459f762235c3 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:29.305580 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13141a28-f02a-4709-b5d2-e05f3c102f44 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:29.306277 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13141a28-f02a-4709-b5d2-e05f3c102f44 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:29.306558 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13141a28-f02a-4709-b5d2-e05f3c102f44 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:29.306830 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13141a28-f02a-4709-b5d2-e05f3c102f44 admin admin] PATH_INFO: `/v3/projects/e2d267b00bbf4cf8948c6d495c7fe4f5/users/eae600c2a26b437ba34f338c7239fc70/roles/a6e562cdc96c4be691d638990f8403fd` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:29.326266 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13141a28-f02a-4709-b5d2-e05f3c102f44 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e2d267b00bbf4cf8948c6d495c7fe4f5, user_id=eae600c2a26b437ba34f338c7239fc70, role_id=a6e562cdc96c4be691d638990f8403fd)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:29.330477 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13141a28-f02a-4709-b5d2-e05f3c102f44 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:29.341393 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4e2d224-09f3-4fb1-985f-6d850c8470ba admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:29.346129 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1399/2790] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:42:29 2022] PUT /identity/v3/projects/e2d267b00bbf4cf8948c6d495c7fe4f5/users/eae600c2a26b437ba34f338c7239fc70/roles/a6e562cdc96c4be691d638990f8403fd => generated 0 bytes in 62 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 21:42:29.347151 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13141a28-f02a-4709-b5d2-e05f3c102f44 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:29.348947 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-13141a28-f02a-4709-b5d2-e05f3c102f44 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:29.401717 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-854fc773-5222-40d8-bbfb-1e9a5f5d3ce3 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c63a4eaf48ad43529f0aa328f2ebfd29', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:29.402409 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-854fc773-5222-40d8-bbfb-1e9a5f5d3ce3 service glance] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:29.402700 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-854fc773-5222-40d8-bbfb-1e9a5f5d3ce3 service glance] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:29.403014 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-854fc773-5222-40d8-bbfb-1e9a5f5d3ce3 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:29.431745 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13141a28-f02a-4709-b5d2-e05f3c102f44 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:29.438595 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2063bc1-92c6-4864-b9b5-cc62473bf06b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:29.439300 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2063bc1-92c6-4864-b9b5-cc62473bf06b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:29.439583 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2063bc1-92c6-4864-b9b5-cc62473bf06b admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:29.439856 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2063bc1-92c6-4864-b9b5-cc62473bf06b admin admin] PATH_INFO: `/v3/projects/5f7b23fc68b7490a9bf7e504704b541a` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:29.440993 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2063bc1-92c6-4864-b9b5-cc62473bf06b admin admin] RBAC: Authorizing `identity:delete_project(project_id=5f7b23fc68b7490a9bf7e504704b541a)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:29.442943 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2063bc1-92c6-4864-b9b5-cc62473bf06b admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:29.473001 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-854fc773-5222-40d8-bbfb-1e9a5f5d3ce3 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:29.474483 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-854fc773-5222-40d8-bbfb-1e9a5f5d3ce3 service glance] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:29.477779 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=97,keystone:INSERT=2,keystone:DELETE=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:42:29.484716 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1392/2791] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:42:29 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 230 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 21:42:29.485677 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-854fc773-5222-40d8-bbfb-1e9a5f5d3ce3 service glance] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:29.501458 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-854fc773-5222-40d8-bbfb-1e9a5f5d3ce3 service glance] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:29.505262 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-f2063bc1-92c6-4864-b9b5-cc62473bf06b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5f7b23fc68b7490a9bf7e504704b541a', 'request_id': 'req-f2063bc1-92c6-4864-b9b5-cc62473bf06b'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:29.512164 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-f2063bc1-92c6-4864-b9b5-cc62473bf06b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5f7b23fc68b7490a9bf7e504704b541a', 'request_id': 'req-f2063bc1-92c6-4864-b9b5-cc62473bf06b'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:29.516204 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:42:29.522249 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-f2063bc1-92c6-4864-b9b5-cc62473bf06b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5f7b23fc68b7490a9bf7e504704b541a', 'request_id': 'req-f2063bc1-92c6-4864-b9b5-cc62473bf06b'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:29.540162 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-f2063bc1-92c6-4864-b9b5-cc62473bf06b admin admin] The token cache is being invalidate because project 5f7b23fc68b7490a9bf7e504704b541a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:42:29.540440 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-f2063bc1-92c6-4864-b9b5-cc62473bf06b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:29.541257 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-f2063bc1-92c6-4864-b9b5-cc62473bf06b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:29.543622 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1400/2792] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:42:29 2022] DELETE /identity/v3/projects/5f7b23fc68b7490a9bf7e504704b541a => generated 0 bytes in 197 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:42:29.549418 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49c2082c-0621-4387-b611-b55f618d5d9e None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:29.549709 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49c2082c-0621-4387-b611-b55f618d5d9e None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:29.550020 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49c2082c-0621-4387-b611-b55f618d5d9e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:29.572218 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-49c2082c-0621-4387-b611-b55f618d5d9e None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:42:29.573655 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-49c2082c-0621-4387-b611-b55f618d5d9e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:29.582822 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcbb34c6-7d95-41e9-bbea-b6093401d001 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:29.583521 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcbb34c6-7d95-41e9-bbea-b6093401d001 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:29.583880 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcbb34c6-7d95-41e9-bbea-b6093401d001 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:29.584252 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcbb34c6-7d95-41e9-bbea-b6093401d001 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:29.584910 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcbb34c6-7d95-41e9-bbea-b6093401d001 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:29.586544 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcbb34c6-7d95-41e9-bbea-b6093401d001 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:29.602073 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1393/2793] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:42:29 2022] GET /identity/v3/roles => generated 1420 bytes in 117 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:42:29.603097 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcbb34c6-7d95-41e9-bbea-b6093401d001 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:29.605226 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-fcbb34c6-7d95-41e9-bbea-b6093401d001 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:29.620771 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcbb34c6-7d95-41e9-bbea-b6093401d001 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:29.627095 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47b9eaf1-c5a4-43e6-8322-18010dd69b31 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:29.628518 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47b9eaf1-c5a4-43e6-8322-18010dd69b31 admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:29.628875 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47b9eaf1-c5a4-43e6-8322-18010dd69b31 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:29.629222 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47b9eaf1-c5a4-43e6-8322-18010dd69b31 admin admin] PATH_INFO: `/v3/projects/e2d267b00bbf4cf8948c6d495c7fe4f5/users/eae600c2a26b437ba34f338c7239fc70/roles/a6e562cdc96c4be691d638990f8403fd` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:29.631719 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47b9eaf1-c5a4-43e6-8322-18010dd69b31 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e2d267b00bbf4cf8948c6d495c7fe4f5, user_id=eae600c2a26b437ba34f338c7239fc70, role_id=a6e562cdc96c4be691d638990f8403fd)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:29.634620 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47b9eaf1-c5a4-43e6-8322-18010dd69b31 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:29.648398 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1394/2794] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:42:29 2022] PUT /identity/v3/projects/e2d267b00bbf4cf8948c6d495c7fe4f5/users/eae600c2a26b437ba34f338c7239fc70/roles/a6e562cdc96c4be691d638990f8403fd => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:42:29.655496 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1401/2795] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:42:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:42:29.659297 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61370488-0c9b-418a-9a90-0483f20adc03 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:29.659588 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61370488-0c9b-418a-9a90-0483f20adc03 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:29.659866 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61370488-0c9b-418a-9a90-0483f20adc03 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:29.666977 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e74c5c6-28fd-4eb8-ad4b-88e5bd3fb022 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:29.667278 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e74c5c6-28fd-4eb8-ad4b-88e5bd3fb022 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:29.667619 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e74c5c6-28fd-4eb8-ad4b-88e5bd3fb022 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:29.690618 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-7e74c5c6-28fd-4eb8-ad4b-88e5bd3fb022 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:42:29.691960 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-7e74c5c6-28fd-4eb8-ad4b-88e5bd3fb022 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:29.701512 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-61370488-0c9b-418a-9a90-0483f20adc03 None None] MFA Rules not processed for user `eae600c2a26b437ba34f338c7239fc70`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:42:29.704183 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-61370488-0c9b-418a-9a90-0483f20adc03 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:29.771972 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1402/2796] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:42:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:42:29.776160 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e74c5c6-28fd-4eb8-ad4b-88e5bd3fb022 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:29.787908 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e74c5c6-28fd-4eb8-ad4b-88e5bd3fb022 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:29.792781 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9cd9eff-4d33-4e3a-a0aa-df4f29954c18 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:29.793449 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9cd9eff-4d33-4e3a-a0aa-df4f29954c18 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:29.793805 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9cd9eff-4d33-4e3a-a0aa-df4f29954c18 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:29.794092 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9cd9eff-4d33-4e3a-a0aa-df4f29954c18 admin admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:29.794927 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9cd9eff-4d33-4e3a-a0aa-df4f29954c18 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:42:29.795238 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9cd9eff-4d33-4e3a-a0aa-df4f29954c18 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:29.798218 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9cd9eff-4d33-4e3a-a0aa-df4f29954c18 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:29.806230 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1395/2797] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:42:29 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 21:42:29.816822 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1403/2798] 10.222.0.90 () {58 vars in 1153 bytes} [Mon Aug 29 21:42:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:42:29.817397 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61370488-0c9b-418a-9a90-0483f20adc03 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:29.818802 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9cd9eff-4d33-4e3a-a0aa-df4f29954c18 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:29.821526 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-e9cd9eff-4d33-4e3a-a0aa-df4f29954c18 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:29.829945 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61370488-0c9b-418a-9a90-0483f20adc03 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:29.835321 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98d417a0-be04-4f5b-922d-e8d98199a619 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:29.835938 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98d417a0-be04-4f5b-922d-e8d98199a619 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:29.836222 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98d417a0-be04-4f5b-922d-e8d98199a619 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:29.836499 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98d417a0-be04-4f5b-922d-e8d98199a619 admin admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:29.837155 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98d417a0-be04-4f5b-922d-e8d98199a619 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:29.839115 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98d417a0-be04-4f5b-922d-e8d98199a619 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:29.882560 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1396/2799] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:42:29 2022] POST /identity/v3/projects => generated 379 bytes in 66 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:42:29.887247 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98d417a0-be04-4f5b-922d-e8d98199a619 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:29.899788 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:DELETE=6,keystone:SELECT=93,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:42:29.920477 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9cd9eff-4d33-4e3a-a0aa-df4f29954c18 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:29.984272 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98d417a0-be04-4f5b-922d-e8d98199a619 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:29.986382 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da0653a5-6386-4de0-aa4e-3f9d40f48495 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dc226f7ddcb49ecacb51ce57fdfc11b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:29.987078 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da0653a5-6386-4de0-aa4e-3f9d40f48495 service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:29.987444 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da0653a5-6386-4de0-aa4e-3f9d40f48495 service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:29.987772 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da0653a5-6386-4de0-aa4e-3f9d40f48495 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:29.990098 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72fc8c96-9216-4d8f-a963-e6b57ef549c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:29.990851 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72fc8c96-9216-4d8f-a963-e6b57ef549c2 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:29.991138 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72fc8c96-9216-4d8f-a963-e6b57ef549c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:29.991416 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72fc8c96-9216-4d8f-a963-e6b57ef549c2 admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:29.992324 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72fc8c96-9216-4d8f-a963-e6b57ef549c2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:29.995793 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72fc8c96-9216-4d8f-a963-e6b57ef549c2 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:30.005349 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:42:30.026400 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1397/2800] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:42:29 2022] POST /identity/v3/users => generated 290 bytes in 140 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 21:42:30.032239 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72fc8c96-9216-4d8f-a963-e6b57ef549c2 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:30.062166 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72fc8c96-9216-4d8f-a963-e6b57ef549c2 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:30.071381 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-717ebd83-5d3b-405c-bd0d-b2ff3c8cc5f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:30.072031 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-717ebd83-5d3b-405c-bd0d-b2ff3c8cc5f7 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:30.073922 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-717ebd83-5d3b-405c-bd0d-b2ff3c8cc5f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:30.074498 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-717ebd83-5d3b-405c-bd0d-b2ff3c8cc5f7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:30.075501 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-717ebd83-5d3b-405c-bd0d-b2ff3c8cc5f7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:30.078539 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-717ebd83-5d3b-405c-bd0d-b2ff3c8cc5f7 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:30.084241 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da0653a5-6386-4de0-aa4e-3f9d40f48495 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:30.085858 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da0653a5-6386-4de0-aa4e-3f9d40f48495 service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:30.092054 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1398/2801] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:42:30 2022] GET /identity/v3/roles => generated 1420 bytes in 61 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:42:30.098614 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-717ebd83-5d3b-405c-bd0d-b2ff3c8cc5f7 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:30.100575 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1404/2802] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:42:29 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 282 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:42:30.112257 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-717ebd83-5d3b-405c-bd0d-b2ff3c8cc5f7 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:30.118666 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1641cf6-aa84-46d0-bc13-8a16ef4e4a13 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:30.119471 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1641cf6-aa84-46d0-bc13-8a16ef4e4a13 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:30.119839 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1641cf6-aa84-46d0-bc13-8a16ef4e4a13 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:30.120204 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1641cf6-aa84-46d0-bc13-8a16ef4e4a13 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:30.120744 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1641cf6-aa84-46d0-bc13-8a16ef4e4a13 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:30.122395 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1641cf6-aa84-46d0-bc13-8a16ef4e4a13 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:30.149962 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1399/2803] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:42:30 2022] GET /identity/v3/roles => generated 1420 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:42:30.156204 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da0653a5-6386-4de0-aa4e-3f9d40f48495 service cinder] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:30.168444 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da0653a5-6386-4de0-aa4e-3f9d40f48495 service cinder] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:30.173681 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57cb6a7b-1b41-433b-a358-d823beb9f053 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:30.179425 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57cb6a7b-1b41-433b-a358-d823beb9f053 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:30.179425 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57cb6a7b-1b41-433b-a358-d823beb9f053 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:30.179425 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57cb6a7b-1b41-433b-a358-d823beb9f053 admin admin] PATH_INFO: `/v3/projects/1ba68826bbc24418aabf0c66294578ca/users/7dceb7dc857348fda46f501288144bde/roles/02e242d5520740d4b77a5f4008c7eb57` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:30.180494 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1641cf6-aa84-46d0-bc13-8a16ef4e4a13 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:30.191628 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1641cf6-aa84-46d0-bc13-8a16ef4e4a13 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:30.197927 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-444e2508-30b8-47bf-9782-a486d870fe71 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:30.198569 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-444e2508-30b8-47bf-9782-a486d870fe71 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:30.198858 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-444e2508-30b8-47bf-9782-a486d870fe71 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:30.199139 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-444e2508-30b8-47bf-9782-a486d870fe71 admin admin] PATH_INFO: `/v3/users/b4a9327b5706420881f989160b1c755b` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:30.199533 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57cb6a7b-1b41-433b-a358-d823beb9f053 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1ba68826bbc24418aabf0c66294578ca, user_id=7dceb7dc857348fda46f501288144bde, role_id=02e242d5520740d4b77a5f4008c7eb57)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:30.200618 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-444e2508-30b8-47bf-9782-a486d870fe71 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b4a9327b5706420881f989160b1c755b)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:30.203398 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-444e2508-30b8-47bf-9782-a486d870fe71 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:30.204152 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57cb6a7b-1b41-433b-a358-d823beb9f053 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:30.222960 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1405/2804] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:42:30 2022] PUT /identity/v3/projects/1ba68826bbc24418aabf0c66294578ca/users/7dceb7dc857348fda46f501288144bde/roles/02e242d5520740d4b77a5f4008c7eb57 => generated 0 bytes in 69 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:42:30.228138 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b09bfcf0-7420-4d19-bff0-3c3d66ee1879 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:30.228419 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b09bfcf0-7420-4d19-bff0-3c3d66ee1879 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:30.228694 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b09bfcf0-7420-4d19-bff0-3c3d66ee1879 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:30.255938 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-b09bfcf0-7420-4d19-bff0-3c3d66ee1879 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:42:30.257409 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-b09bfcf0-7420-4d19-bff0-3c3d66ee1879 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:30.281711 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-444e2508-30b8-47bf-9782-a486d870fe71 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b4a9327b5706420881f989160b1c755b'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:30.290660 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-444e2508-30b8-47bf-9782-a486d870fe71 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b4a9327b5706420881f989160b1c755b'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:30.299742 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-444e2508-30b8-47bf-9782-a486d870fe71 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b4a9327b5706420881f989160b1c755b'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:30.300573 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-444e2508-30b8-47bf-9782-a486d870fe71 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b4a9327b5706420881f989160b1c755b'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:30.301212 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-444e2508-30b8-47bf-9782-a486d870fe71 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b4a9327b5706420881f989160b1c755b'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:30.322795 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1400/2805] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:42:30 2022] DELETE /identity/v3/users/b4a9327b5706420881f989160b1c755b => generated 0 bytes in 143 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:42:30.327549 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43e95052-40bf-4212-be8f-fc05da2d1afe None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:30.327888 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43e95052-40bf-4212-be8f-fc05da2d1afe None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:30.328187 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43e95052-40bf-4212-be8f-fc05da2d1afe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:30.331996 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1406/2806] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:42:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:42:30.343499 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b09bfcf0-7420-4d19-bff0-3c3d66ee1879 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:30.345883 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-b09bfcf0-7420-4d19-bff0-3c3d66ee1879 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:30.352950 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-43e95052-40bf-4212-be8f-fc05da2d1afe None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:42:30.354250 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-43e95052-40bf-4212-be8f-fc05da2d1afe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:30.381475 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:42:30.420098 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b09bfcf0-7420-4d19-bff0-3c3d66ee1879 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:30.434221 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1401/2807] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:42:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 21:42:30.446464 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43e95052-40bf-4212-be8f-fc05da2d1afe None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:30.449649 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-43e95052-40bf-4212-be8f-fc05da2d1afe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:30.478559 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ef192e7-e0e8-4114-87a6-9966ab1767f8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:30.479175 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ef192e7-e0e8-4114-87a6-9966ab1767f8 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:30.479447 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ef192e7-e0e8-4114-87a6-9966ab1767f8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:30.479713 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ef192e7-e0e8-4114-87a6-9966ab1767f8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:30.489270 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ef192e7-e0e8-4114-87a6-9966ab1767f8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:30.490803 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ef192e7-e0e8-4114-87a6-9966ab1767f8 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:30.501944 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:INSERT=4,keystone:SELECT=86,keystone:DELETE=10 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:42:30.503040 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1407/2808] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:42:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 161 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:42:30.520024 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43e95052-40bf-4212-be8f-fc05da2d1afe None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:30.523693 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8d05d8e-bd12-40ea-ba91-3aa1900b3e0d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:30.524305 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8d05d8e-bd12-40ea-ba91-3aa1900b3e0d service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:30.528664 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8d05d8e-bd12-40ea-ba91-3aa1900b3e0d service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:30.528976 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8d05d8e-bd12-40ea-ba91-3aa1900b3e0d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:30.539440 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8d05d8e-bd12-40ea-ba91-3aa1900b3e0d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:30.540869 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8d05d8e-bd12-40ea-ba91-3aa1900b3e0d service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:30.554370 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1402/2809] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:42:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 21:42:30.887303 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ef192e7-e0e8-4114-87a6-9966ab1767f8 service neutron] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:30.888947 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-7ef192e7-e0e8-4114-87a6-9966ab1767f8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:30.913858 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ef192e7-e0e8-4114-87a6-9966ab1767f8 service neutron] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:30.922301 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1639de7-46d8-4fa3-8d33-d893fcdbe7b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:30.923202 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1639de7-46d8-4fa3-8d33-d893fcdbe7b7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:30.923450 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1639de7-46d8-4fa3-8d33-d893fcdbe7b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:30.923704 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1639de7-46d8-4fa3-8d33-d893fcdbe7b7 admin admin] PATH_INFO: `/v3/projects/d360668f09044c4bb9c74b45e837e4e0` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:30.925198 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1639de7-46d8-4fa3-8d33-d893fcdbe7b7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d360668f09044c4bb9c74b45e837e4e0)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:30.928691 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1639de7-46d8-4fa3-8d33-d893fcdbe7b7 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:30.992557 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-c1639de7-46d8-4fa3-8d33-d893fcdbe7b7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd360668f09044c4bb9c74b45e837e4e0', 'request_id': 'req-c1639de7-46d8-4fa3-8d33-d893fcdbe7b7'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:30.999996 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-c1639de7-46d8-4fa3-8d33-d893fcdbe7b7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd360668f09044c4bb9c74b45e837e4e0', 'request_id': 'req-c1639de7-46d8-4fa3-8d33-d893fcdbe7b7'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:31.007133 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-c1639de7-46d8-4fa3-8d33-d893fcdbe7b7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd360668f09044c4bb9c74b45e837e4e0', 'request_id': 'req-c1639de7-46d8-4fa3-8d33-d893fcdbe7b7'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:31.022310 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-c1639de7-46d8-4fa3-8d33-d893fcdbe7b7 admin admin] The token cache is being invalidate because project d360668f09044c4bb9c74b45e837e4e0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:42:31.022608 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-c1639de7-46d8-4fa3-8d33-d893fcdbe7b7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:31.023240 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-c1639de7-46d8-4fa3-8d33-d893fcdbe7b7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:31.024956 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1408/2810] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:42:30 2022] DELETE /identity/v3/projects/d360668f09044c4bb9c74b45e837e4e0 => generated 0 bytes in 138 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:42:31.028656 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8d05d8e-bd12-40ea-ba91-3aa1900b3e0d service neutron] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:31.040044 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8d05d8e-bd12-40ea-ba91-3aa1900b3e0d service neutron] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:31.106971 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1108f194-0f9f-4609-aca5-d7c2f1a0abd4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:31.107763 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1108f194-0f9f-4609-aca5-d7c2f1a0abd4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:31.108731 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1108f194-0f9f-4609-aca5-d7c2f1a0abd4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:31.109465 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1108f194-0f9f-4609-aca5-d7c2f1a0abd4 admin admin] PATH_INFO: `/v3/users/eae600c2a26b437ba34f338c7239fc70` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:31.110543 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1108f194-0f9f-4609-aca5-d7c2f1a0abd4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=eae600c2a26b437ba34f338c7239fc70)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:31.113082 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1108f194-0f9f-4609-aca5-d7c2f1a0abd4 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:31.182568 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-1108f194-0f9f-4609-aca5-d7c2f1a0abd4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'eae600c2a26b437ba34f338c7239fc70'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:31.192553 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-1108f194-0f9f-4609-aca5-d7c2f1a0abd4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'eae600c2a26b437ba34f338c7239fc70'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:31.204873 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-1108f194-0f9f-4609-aca5-d7c2f1a0abd4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'eae600c2a26b437ba34f338c7239fc70'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:31.206245 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-1108f194-0f9f-4609-aca5-d7c2f1a0abd4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'eae600c2a26b437ba34f338c7239fc70'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:31.207559 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-1108f194-0f9f-4609-aca5-d7c2f1a0abd4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'eae600c2a26b437ba34f338c7239fc70'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:31.242718 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1403/2811] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:42:31 2022] DELETE /identity/v3/users/eae600c2a26b437ba34f338c7239fc70 => generated 0 bytes in 214 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:42:31.902883 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1639de7-46d8-4fa3-8d33-d893fcdbe7b7 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:31.904775 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-c1639de7-46d8-4fa3-8d33-d893fcdbe7b7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:31.926672 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1639de7-46d8-4fa3-8d33-d893fcdbe7b7 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:32.019132 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4adc11a-5b14-4c8b-bd0f-7d74f9c20aef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:32.020044 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4adc11a-5b14-4c8b-bd0f-7d74f9c20aef admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:32.020325 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4adc11a-5b14-4c8b-bd0f-7d74f9c20aef admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:32.020603 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4adc11a-5b14-4c8b-bd0f-7d74f9c20aef admin admin] PATH_INFO: `/v3/projects/e2d267b00bbf4cf8948c6d495c7fe4f5` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:32.021836 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4adc11a-5b14-4c8b-bd0f-7d74f9c20aef admin admin] RBAC: Authorizing `identity:delete_project(project_id=e2d267b00bbf4cf8948c6d495c7fe4f5)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:32.024658 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4adc11a-5b14-4c8b-bd0f-7d74f9c20aef admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:32.042872 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:42:32.115037 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-f4adc11a-5b14-4c8b-bd0f-7d74f9c20aef admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e2d267b00bbf4cf8948c6d495c7fe4f5', 'request_id': 'req-f4adc11a-5b14-4c8b-bd0f-7d74f9c20aef'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:32.136817 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-f4adc11a-5b14-4c8b-bd0f-7d74f9c20aef admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e2d267b00bbf4cf8948c6d495c7fe4f5', 'request_id': 'req-f4adc11a-5b14-4c8b-bd0f-7d74f9c20aef'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:32.145280 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-f4adc11a-5b14-4c8b-bd0f-7d74f9c20aef admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e2d267b00bbf4cf8948c6d495c7fe4f5', 'request_id': 'req-f4adc11a-5b14-4c8b-bd0f-7d74f9c20aef'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:32.162855 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-f4adc11a-5b14-4c8b-bd0f-7d74f9c20aef admin admin] The token cache is being invalidate because project e2d267b00bbf4cf8948c6d495c7fe4f5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:42:32.163185 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-f4adc11a-5b14-4c8b-bd0f-7d74f9c20aef admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:32.163814 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-f4adc11a-5b14-4c8b-bd0f-7d74f9c20aef admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:32.166152 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1409/2812] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:42:31 2022] DELETE /identity/v3/projects/e2d267b00bbf4cf8948c6d495c7fe4f5 => generated 0 bytes in 264 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:42:32.178064 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22eb1d0b-83d8-48a5-9135-021d872b0da0 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:32.178638 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22eb1d0b-83d8-48a5-9135-021d872b0da0 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:32.178638 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22eb1d0b-83d8-48a5-9135-021d872b0da0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:32.199690 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-22eb1d0b-83d8-48a5-9135-021d872b0da0 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:42:32.202982 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-22eb1d0b-83d8-48a5-9135-021d872b0da0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:32.289881 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1404/2813] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:42:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:42:32.297488 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5d9aeba-15b6-46b3-bdba-1409c5c27e10 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:32.297766 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5d9aeba-15b6-46b3-bdba-1409c5c27e10 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:32.298125 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5d9aeba-15b6-46b3-bdba-1409c5c27e10 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:32.351424 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-c5d9aeba-15b6-46b3-bdba-1409c5c27e10 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:42:32.352878 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-c5d9aeba-15b6-46b3-bdba-1409c5c27e10 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:32.370380 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1410/2814] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:42:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 72 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:42:32.376301 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22eb1d0b-83d8-48a5-9135-021d872b0da0 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:32.381735 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:42:32.408770 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22eb1d0b-83d8-48a5-9135-021d872b0da0 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:32.427059 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88781ce6-24bf-4ade-a482-5ca651be49ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:32.427616 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88781ce6-24bf-4ade-a482-5ca651be49ce admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:32.428173 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88781ce6-24bf-4ade-a482-5ca651be49ce admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:32.428447 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88781ce6-24bf-4ade-a482-5ca651be49ce admin admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:32.429872 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88781ce6-24bf-4ade-a482-5ca651be49ce admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:42:32.430280 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88781ce6-24bf-4ade-a482-5ca651be49ce admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:32.431865 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88781ce6-24bf-4ade-a482-5ca651be49ce admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:32.458750 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1405/2815] 10.222.0.90 () {58 vars in 1153 bytes} [Mon Aug 29 21:42:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:42:32.462928 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5d9aeba-15b6-46b3-bdba-1409c5c27e10 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:32.480534 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5d9aeba-15b6-46b3-bdba-1409c5c27e10 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:32.488937 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1894ec4c-eac1-4d5c-add1-25df2ac2ac21 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:32.489574 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1894ec4c-eac1-4d5c-add1-25df2ac2ac21 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:32.490269 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1894ec4c-eac1-4d5c-add1-25df2ac2ac21 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:32.490269 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1894ec4c-eac1-4d5c-add1-25df2ac2ac21 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:32.491040 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1894ec4c-eac1-4d5c-add1-25df2ac2ac21 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:32.493261 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1894ec4c-eac1-4d5c-add1-25df2ac2ac21 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:32.534900 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1411/2816] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:42:32 2022] POST /identity/v3/projects => generated 387 bytes in 67 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:42:32.536494 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88781ce6-24bf-4ade-a482-5ca651be49ce admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:32.552877 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88781ce6-24bf-4ade-a482-5ca651be49ce admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:32.652728 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a392359-67c1-48f0-aedf-6db821f524bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:32.653498 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a392359-67c1-48f0-aedf-6db821f524bb admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:32.653931 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a392359-67c1-48f0-aedf-6db821f524bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:32.654303 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a392359-67c1-48f0-aedf-6db821f524bb admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:32.655192 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a392359-67c1-48f0-aedf-6db821f524bb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:32.657295 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a392359-67c1-48f0-aedf-6db821f524bb admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:32.693310 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1406/2817] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:42:32 2022] POST /identity/v3/users => generated 294 bytes in 158 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:42:32.697449 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1894ec4c-eac1-4d5c-add1-25df2ac2ac21 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:32.711158 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1894ec4c-eac1-4d5c-add1-25df2ac2ac21 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:32.716573 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f05c6ea9-503d-4185-8f1c-cdec19a42ac9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:32.717524 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f05c6ea9-503d-4185-8f1c-cdec19a42ac9 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:32.717998 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f05c6ea9-503d-4185-8f1c-cdec19a42ac9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:32.718345 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f05c6ea9-503d-4185-8f1c-cdec19a42ac9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:32.719307 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f05c6ea9-503d-4185-8f1c-cdec19a42ac9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:32.721360 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f05c6ea9-503d-4185-8f1c-cdec19a42ac9 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:32.735945 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1412/2818] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:42:32 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:42:32.739589 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a392359-67c1-48f0-aedf-6db821f524bb admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:32.751898 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a392359-67c1-48f0-aedf-6db821f524bb admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:32.757180 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5a35583-a8ba-4ded-a945-f650a06acdee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:32.757881 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5a35583-a8ba-4ded-a945-f650a06acdee admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:32.758166 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5a35583-a8ba-4ded-a945-f650a06acdee admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:32.758440 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5a35583-a8ba-4ded-a945-f650a06acdee admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:32.759104 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5a35583-a8ba-4ded-a945-f650a06acdee admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:32.760865 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5a35583-a8ba-4ded-a945-f650a06acdee admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:32.773823 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1407/2819] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:42:32 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:42:32.778676 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f05c6ea9-503d-4185-8f1c-cdec19a42ac9 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:32.789620 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f05c6ea9-503d-4185-8f1c-cdec19a42ac9 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:32.794719 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11aa52c4-856d-4789-a8b8-9b206670059c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:32.795756 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11aa52c4-856d-4789-a8b8-9b206670059c admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:32.796040 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11aa52c4-856d-4789-a8b8-9b206670059c admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:32.796315 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11aa52c4-856d-4789-a8b8-9b206670059c admin admin] PATH_INFO: `/v3/projects/be24cb3c41b34759bfaa60a1f0cead52/users/d4da2a64dabc44c7aae3fd96fcfe0d12/roles/02e242d5520740d4b77a5f4008c7eb57` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:32.819222 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11aa52c4-856d-4789-a8b8-9b206670059c admin admin] RBAC: Authorizing `identity:create_grant(project_id=be24cb3c41b34759bfaa60a1f0cead52, user_id=d4da2a64dabc44c7aae3fd96fcfe0d12, role_id=02e242d5520740d4b77a5f4008c7eb57)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:32.822289 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11aa52c4-856d-4789-a8b8-9b206670059c admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:32.843248 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1413/2820] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:42:32 2022] PUT /identity/v3/projects/be24cb3c41b34759bfaa60a1f0cead52/users/d4da2a64dabc44c7aae3fd96fcfe0d12/roles/02e242d5520740d4b77a5f4008c7eb57 => generated 0 bytes in 65 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 21:42:32.855928 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35b8d69d-5c43-4a79-9434-d23817f8a985 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:32.856271 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35b8d69d-5c43-4a79-9434-d23817f8a985 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:32.856490 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35b8d69d-5c43-4a79-9434-d23817f8a985 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:32.901827 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-35b8d69d-5c43-4a79-9434-d23817f8a985 None None] MFA Rules not processed for user `d4da2a64dabc44c7aae3fd96fcfe0d12`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:42:32.905739 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-35b8d69d-5c43-4a79-9434-d23817f8a985 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:33.014857 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1408/2821] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:42:32 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:42:33.019710 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11aa52c4-856d-4789-a8b8-9b206670059c admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:33.119310 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11aa52c4-856d-4789-a8b8-9b206670059c admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:33.128715 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b3358ca-6fba-4ba0-aa8c-333c4e710ec9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:33.130227 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b3358ca-6fba-4ba0-aa8c-333c4e710ec9 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:33.130357 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b3358ca-6fba-4ba0-aa8c-333c4e710ec9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:33.130759 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b3358ca-6fba-4ba0-aa8c-333c4e710ec9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:33.131859 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b3358ca-6fba-4ba0-aa8c-333c4e710ec9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:33.135230 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b3358ca-6fba-4ba0-aa8c-333c4e710ec9 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:33.193445 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1414/2822] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:42:33 2022] POST /identity/v3/projects => generated 387 bytes in 170 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:42:33.194814 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35b8d69d-5c43-4a79-9434-d23817f8a985 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:33.209353 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35b8d69d-5c43-4a79-9434-d23817f8a985 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:33.295865 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:42:33.306823 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-269a77e9-ed9b-4505-8315-9335546de113 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:33.307466 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-269a77e9-ed9b-4505-8315-9335546de113 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:33.307755 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-269a77e9-ed9b-4505-8315-9335546de113 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:33.308032 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-269a77e9-ed9b-4505-8315-9335546de113 admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:33.308679 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-269a77e9-ed9b-4505-8315-9335546de113 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:33.310829 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-269a77e9-ed9b-4505-8315-9335546de113 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:33.337856 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1409/2823] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:42:33 2022] POST /identity/v3/users => generated 294 bytes in 142 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:42:33.340697 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b3358ca-6fba-4ba0-aa8c-333c4e710ec9 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:33.351918 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b3358ca-6fba-4ba0-aa8c-333c4e710ec9 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:33.356853 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74c0c5b2-aead-4afc-bd75-a72fa1919ab2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:33.357829 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74c0c5b2-aead-4afc-bd75-a72fa1919ab2 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:33.358589 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74c0c5b2-aead-4afc-bd75-a72fa1919ab2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:33.359228 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74c0c5b2-aead-4afc-bd75-a72fa1919ab2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:33.362178 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74c0c5b2-aead-4afc-bd75-a72fa1919ab2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:33.364168 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74c0c5b2-aead-4afc-bd75-a72fa1919ab2 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:33.381850 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1415/2824] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:42:33 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:42:33.383955 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-269a77e9-ed9b-4505-8315-9335546de113 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:33.404177 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-269a77e9-ed9b-4505-8315-9335546de113 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:33.414785 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b46d07f-474a-403a-a1a8-9efbab97aa83 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:33.416039 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b46d07f-474a-403a-a1a8-9efbab97aa83 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:33.416039 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b46d07f-474a-403a-a1a8-9efbab97aa83 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:33.416302 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b46d07f-474a-403a-a1a8-9efbab97aa83 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:33.416753 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b46d07f-474a-403a-a1a8-9efbab97aa83 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:33.418724 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b46d07f-474a-403a-a1a8-9efbab97aa83 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:33.437322 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1410/2825] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:42:33 2022] GET /identity/v3/roles => generated 1420 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 29 21:42:33.442197 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74c0c5b2-aead-4afc-bd75-a72fa1919ab2 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:33.454491 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74c0c5b2-aead-4afc-bd75-a72fa1919ab2 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:33.462200 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c42e7d4-eda2-497d-8d04-f3b82e93a577 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:33.462890 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c42e7d4-eda2-497d-8d04-f3b82e93a577 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:33.463172 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c42e7d4-eda2-497d-8d04-f3b82e93a577 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:33.463446 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c42e7d4-eda2-497d-8d04-f3b82e93a577 admin admin] PATH_INFO: `/v3/projects/38925e4eaa5041a5882f8c6166dff34e/users/de7588262bee4141bf2f1de2873664db/roles/02e242d5520740d4b77a5f4008c7eb57` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:33.466427 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:42:33.486379 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c42e7d4-eda2-497d-8d04-f3b82e93a577 admin admin] RBAC: Authorizing `identity:create_grant(project_id=38925e4eaa5041a5882f8c6166dff34e, user_id=de7588262bee4141bf2f1de2873664db, role_id=02e242d5520740d4b77a5f4008c7eb57)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:33.489315 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c42e7d4-eda2-497d-8d04-f3b82e93a577 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:33.541034 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1416/2826] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:42:33 2022] PUT /identity/v3/projects/38925e4eaa5041a5882f8c6166dff34e/users/de7588262bee4141bf2f1de2873664db/roles/02e242d5520740d4b77a5f4008c7eb57 => generated 0 bytes in 91 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:42:33.543512 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bd7a2f6-864a-48f4-9d27-e832e5eadc86 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:33.543807 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bd7a2f6-864a-48f4-9d27-e832e5eadc86 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:33.544227 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bd7a2f6-864a-48f4-9d27-e832e5eadc86 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:33.596022 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-4bd7a2f6-864a-48f4-9d27-e832e5eadc86 None None] MFA Rules not processed for user `de7588262bee4141bf2f1de2873664db`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:42:33.597412 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-4bd7a2f6-864a-48f4-9d27-e832e5eadc86 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:33.714243 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1411/2827] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:42:33 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 21:42:33.715726 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c42e7d4-eda2-497d-8d04-f3b82e93a577 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:33.821984 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c42e7d4-eda2-497d-8d04-f3b82e93a577 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:33.829212 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c915a27f-a657-4b18-b66d-904ec05b781e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:33.829990 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c915a27f-a657-4b18-b66d-904ec05b781e admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:33.830346 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c915a27f-a657-4b18-b66d-904ec05b781e admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:33.831125 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c915a27f-a657-4b18-b66d-904ec05b781e admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:33.831868 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c915a27f-a657-4b18-b66d-904ec05b781e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:33.834639 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c915a27f-a657-4b18-b66d-904ec05b781e admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:33.896251 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1417/2828] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:42:33 2022] POST /identity/v3/projects => generated 387 bytes in 181 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:42:33.896600 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bd7a2f6-864a-48f4-9d27-e832e5eadc86 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:33.909632 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bd7a2f6-864a-48f4-9d27-e832e5eadc86 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:33.982159 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b12f071-50a9-4831-8798-49b00a3996e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:33.983804 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b12f071-50a9-4831-8798-49b00a3996e1 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:33.984105 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b12f071-50a9-4831-8798-49b00a3996e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:33.984382 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b12f071-50a9-4831-8798-49b00a3996e1 admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:33.985134 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b12f071-50a9-4831-8798-49b00a3996e1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:33.988025 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b12f071-50a9-4831-8798-49b00a3996e1 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:34.022411 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1412/2829] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:42:33 2022] POST /identity/v3/users => generated 300 bytes in 126 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 21:42:34.026556 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c915a27f-a657-4b18-b66d-904ec05b781e admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:34.037551 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c915a27f-a657-4b18-b66d-904ec05b781e admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:34.043486 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c9a9609-9ba2-4209-bfb5-6b3ba417a110 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:34.044116 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c9a9609-9ba2-4209-bfb5-6b3ba417a110 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:34.044421 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c9a9609-9ba2-4209-bfb5-6b3ba417a110 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:34.044683 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c9a9609-9ba2-4209-bfb5-6b3ba417a110 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:34.045426 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c9a9609-9ba2-4209-bfb5-6b3ba417a110 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:34.047235 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c9a9609-9ba2-4209-bfb5-6b3ba417a110 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:34.060245 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1418/2830] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:42:34 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:42:34.068278 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b12f071-50a9-4831-8798-49b00a3996e1 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:34.082630 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b12f071-50a9-4831-8798-49b00a3996e1 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:34.087795 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3edf6cb2-0e79-4157-bd35-801d16a43a83 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:34.088484 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3edf6cb2-0e79-4157-bd35-801d16a43a83 admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:34.088769 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3edf6cb2-0e79-4157-bd35-801d16a43a83 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:34.089195 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3edf6cb2-0e79-4157-bd35-801d16a43a83 admin admin] PATH_INFO: `/v3/projects/d34f3d31b81f4864a868e7eb0f83f736/users/9da169756d1f45ee89cc6e0fad66d34e/roles/a6e562cdc96c4be691d638990f8403fd` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:34.106256 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3edf6cb2-0e79-4157-bd35-801d16a43a83 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d34f3d31b81f4864a868e7eb0f83f736, user_id=9da169756d1f45ee89cc6e0fad66d34e, role_id=a6e562cdc96c4be691d638990f8403fd)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:34.109491 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3edf6cb2-0e79-4157-bd35-801d16a43a83 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:34.136602 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1413/2831] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:42:34 2022] PUT /identity/v3/projects/d34f3d31b81f4864a868e7eb0f83f736/users/9da169756d1f45ee89cc6e0fad66d34e/roles/a6e562cdc96c4be691d638990f8403fd => generated 0 bytes in 69 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:42:34.142134 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c9a9609-9ba2-4209-bfb5-6b3ba417a110 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:34.154802 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c9a9609-9ba2-4209-bfb5-6b3ba417a110 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:34.235563 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6c988f1-e299-4d50-bdd4-177b5840ef62 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:34.236198 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c988f1-e299-4d50-bdd4-177b5840ef62 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:34.236499 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c988f1-e299-4d50-bdd4-177b5840ef62 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:34.236745 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c988f1-e299-4d50-bdd4-177b5840ef62 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:34.237396 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6c988f1-e299-4d50-bdd4-177b5840ef62 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:34.239596 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6c988f1-e299-4d50-bdd4-177b5840ef62 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:34.254701 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1419/2832] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:42:34 2022] GET /identity/v3/roles => generated 1420 bytes in 113 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:42:34.259023 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3edf6cb2-0e79-4157-bd35-801d16a43a83 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:34.272367 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3edf6cb2-0e79-4157-bd35-801d16a43a83 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:34.278787 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ce0015f-8e63-4279-aeb6-df7bd675a0a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:34.282030 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ce0015f-8e63-4279-aeb6-df7bd675a0a7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:34.282030 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ce0015f-8e63-4279-aeb6-df7bd675a0a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:34.282030 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ce0015f-8e63-4279-aeb6-df7bd675a0a7 admin admin] PATH_INFO: `/v3/projects/d34f3d31b81f4864a868e7eb0f83f736/users/9da169756d1f45ee89cc6e0fad66d34e/roles/a6e562cdc96c4be691d638990f8403fd` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:34.283536 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ce0015f-8e63-4279-aeb6-df7bd675a0a7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d34f3d31b81f4864a868e7eb0f83f736, user_id=9da169756d1f45ee89cc6e0fad66d34e, role_id=a6e562cdc96c4be691d638990f8403fd)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:34.286605 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ce0015f-8e63-4279-aeb6-df7bd675a0a7 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:34.301642 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1414/2833] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:42:34 2022] PUT /identity/v3/projects/d34f3d31b81f4864a868e7eb0f83f736/users/9da169756d1f45ee89cc6e0fad66d34e/roles/a6e562cdc96c4be691d638990f8403fd => generated 0 bytes in 43 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:42:34.311833 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06a70040-629c-4207-b419-044a14889640 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:34.312140 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06a70040-629c-4207-b419-044a14889640 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:34.312449 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06a70040-629c-4207-b419-044a14889640 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:34.364223 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-06a70040-629c-4207-b419-044a14889640 None None] MFA Rules not processed for user `9da169756d1f45ee89cc6e0fad66d34e`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:42:34.365998 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-06a70040-629c-4207-b419-044a14889640 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:34.428552 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:42:34.495114 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1420/2834] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:42:34 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:42:34.506526 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ce0015f-8e63-4279-aeb6-df7bd675a0a7 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:34.508661 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-4ce0015f-8e63-4279-aeb6-df7bd675a0a7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:34.542476 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:42:34.666874 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ce0015f-8e63-4279-aeb6-df7bd675a0a7 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:34.677840 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2f6c389-2067-43fd-9275-c8fcd295e14a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dc226f7ddcb49ecacb51ce57fdfc11b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:34.678488 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2f6c389-2067-43fd-9275-c8fcd295e14a service cinder] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:34.678771 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2f6c389-2067-43fd-9275-c8fcd295e14a service cinder] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:34.679041 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2f6c389-2067-43fd-9275-c8fcd295e14a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:34.715113 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2f6c389-2067-43fd-9275-c8fcd295e14a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:34.718983 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2f6c389-2067-43fd-9275-c8fcd295e14a service cinder] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:34.790376 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1415/2835] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:42:34 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 284 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:42:35.610843 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06a70040-629c-4207-b419-044a14889640 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:35.621740 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06a70040-629c-4207-b419-044a14889640 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:35.626468 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75e9b5b1-4cb1-48c1-abd0-11a6759b8a61 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dc226f7ddcb49ecacb51ce57fdfc11b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:35.627029 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75e9b5b1-4cb1-48c1-abd0-11a6759b8a61 service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:35.627307 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75e9b5b1-4cb1-48c1-abd0-11a6759b8a61 service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:35.627577 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75e9b5b1-4cb1-48c1-abd0-11a6759b8a61 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:35.741402 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75e9b5b1-4cb1-48c1-abd0-11a6759b8a61 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:35.744414 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75e9b5b1-4cb1-48c1-abd0-11a6759b8a61 service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:35.762520 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1421/2836] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:42:35 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 153 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:42:36.000770 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16edf65f-8388-450d-8cca-c37d42eee9b7 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:36.001679 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16edf65f-8388-450d-8cca-c37d42eee9b7 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:36.002367 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16edf65f-8388-450d-8cca-c37d42eee9b7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:36.047237 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-16edf65f-8388-450d-8cca-c37d42eee9b7 None None] MFA Rules not processed for user `7dceb7dc857348fda46f501288144bde`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:42:36.048798 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-16edf65f-8388-450d-8cca-c37d42eee9b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:36.148125 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1416/2837] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:42:35 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:42:36.152515 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75e9b5b1-4cb1-48c1-abd0-11a6759b8a61 service cinder] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:36.154028 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-75e9b5b1-4cb1-48c1-abd0-11a6759b8a61 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:36.262503 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75e9b5b1-4cb1-48c1-abd0-11a6759b8a61 service cinder] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:36.268054 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ab3a544-61ad-42c4-a4c3-c19b06f5d787 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:36.268753 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ab3a544-61ad-42c4-a4c3-c19b06f5d787 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:36.269097 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ab3a544-61ad-42c4-a4c3-c19b06f5d787 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:36.269435 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ab3a544-61ad-42c4-a4c3-c19b06f5d787 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:36.270219 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ab3a544-61ad-42c4-a4c3-c19b06f5d787 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:36.272356 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ab3a544-61ad-42c4-a4c3-c19b06f5d787 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:36.322549 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1422/2838] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:42:36 2022] POST /identity/v3/projects => generated 375 bytes in 170 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 21:42:36.323542 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16edf65f-8388-450d-8cca-c37d42eee9b7 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:36.335973 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16edf65f-8388-450d-8cca-c37d42eee9b7 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:36.407031 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e2d9528-576f-4219-8672-3c53ae58551a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:36.407713 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e2d9528-576f-4219-8672-3c53ae58551a admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:36.408010 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e2d9528-576f-4219-8672-3c53ae58551a admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:36.408299 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e2d9528-576f-4219-8672-3c53ae58551a admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:36.409110 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e2d9528-576f-4219-8672-3c53ae58551a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:36.411303 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e2d9528-576f-4219-8672-3c53ae58551a admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:36.435581 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1417/2839] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:42:36 2022] POST /identity/v3/users => generated 294 bytes in 113 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:42:36.439596 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ab3a544-61ad-42c4-a4c3-c19b06f5d787 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:36.451007 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ab3a544-61ad-42c4-a4c3-c19b06f5d787 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:36.456292 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de5a79b6-8d9d-496a-9e5a-fa4aab1abb84 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:36.456904 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de5a79b6-8d9d-496a-9e5a-fa4aab1abb84 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:36.457202 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de5a79b6-8d9d-496a-9e5a-fa4aab1abb84 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:36.457481 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de5a79b6-8d9d-496a-9e5a-fa4aab1abb84 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:36.458165 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de5a79b6-8d9d-496a-9e5a-fa4aab1abb84 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:36.459771 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de5a79b6-8d9d-496a-9e5a-fa4aab1abb84 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:36.472509 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1423/2840] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:42:36 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:42:36.476530 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e2d9528-576f-4219-8672-3c53ae58551a admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:36.487905 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e2d9528-576f-4219-8672-3c53ae58551a admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:36.493264 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba167065-fb6c-4dd5-9bb7-db9ac93bb7a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:36.493983 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba167065-fb6c-4dd5-9bb7-db9ac93bb7a5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:36.494260 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba167065-fb6c-4dd5-9bb7-db9ac93bb7a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:36.494528 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba167065-fb6c-4dd5-9bb7-db9ac93bb7a5 admin admin] PATH_INFO: `/v3/projects/c0cedb3be4f24389b32089be9afb03b1/users/f20085430c7e4d5983c4880fe8bc2b58/roles/a6e562cdc96c4be691d638990f8403fd` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:36.511666 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba167065-fb6c-4dd5-9bb7-db9ac93bb7a5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c0cedb3be4f24389b32089be9afb03b1, user_id=f20085430c7e4d5983c4880fe8bc2b58, role_id=a6e562cdc96c4be691d638990f8403fd)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:36.519576 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba167065-fb6c-4dd5-9bb7-db9ac93bb7a5 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:36.536276 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1418/2841] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:42:36 2022] PUT /identity/v3/projects/c0cedb3be4f24389b32089be9afb03b1/users/f20085430c7e4d5983c4880fe8bc2b58/roles/a6e562cdc96c4be691d638990f8403fd => generated 0 bytes in 60 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:42:36.541135 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de5a79b6-8d9d-496a-9e5a-fa4aab1abb84 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:36.560504 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de5a79b6-8d9d-496a-9e5a-fa4aab1abb84 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:36.634527 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:42:36.656705 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23b85170-1849-4c33-be64-8ad500a0fcd1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:36.657409 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23b85170-1849-4c33-be64-8ad500a0fcd1 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:36.657628 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23b85170-1849-4c33-be64-8ad500a0fcd1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:36.657938 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23b85170-1849-4c33-be64-8ad500a0fcd1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:36.658747 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23b85170-1849-4c33-be64-8ad500a0fcd1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:36.660380 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23b85170-1849-4c33-be64-8ad500a0fcd1 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:36.672055 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1424/2842] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:42:36 2022] GET /identity/v3/roles => generated 1420 bytes in 132 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:42:36.676432 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba167065-fb6c-4dd5-9bb7-db9ac93bb7a5 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:36.688914 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba167065-fb6c-4dd5-9bb7-db9ac93bb7a5 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:36.699398 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91000797-1fb7-437b-b3ac-2737a885eb20 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:36.700401 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91000797-1fb7-437b-b3ac-2737a885eb20 admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:36.700939 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91000797-1fb7-437b-b3ac-2737a885eb20 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:36.701826 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91000797-1fb7-437b-b3ac-2737a885eb20 admin admin] PATH_INFO: `/v3/projects/c0cedb3be4f24389b32089be9afb03b1/users/f20085430c7e4d5983c4880fe8bc2b58/roles/a6e562cdc96c4be691d638990f8403fd` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:36.703916 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91000797-1fb7-437b-b3ac-2737a885eb20 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c0cedb3be4f24389b32089be9afb03b1, user_id=f20085430c7e4d5983c4880fe8bc2b58, role_id=a6e562cdc96c4be691d638990f8403fd)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:36.706626 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91000797-1fb7-437b-b3ac-2737a885eb20 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:36.718326 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1419/2843] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:42:36 2022] PUT /identity/v3/projects/c0cedb3be4f24389b32089be9afb03b1/users/f20085430c7e4d5983c4880fe8bc2b58/roles/a6e562cdc96c4be691d638990f8403fd => generated 0 bytes in 43 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:42:40.705396 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23b85170-1849-4c33-be64-8ad500a0fcd1 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:40.821902 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23b85170-1849-4c33-be64-8ad500a0fcd1 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:40.944224 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-384cf4fb-9bdc-40e3-942b-8dcaffe7fa11 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dc226f7ddcb49ecacb51ce57fdfc11b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:40.945159 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-384cf4fb-9bdc-40e3-942b-8dcaffe7fa11 service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:40.945616 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-384cf4fb-9bdc-40e3-942b-8dcaffe7fa11 service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:40.946089 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-384cf4fb-9bdc-40e3-942b-8dcaffe7fa11 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:41.082127 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-384cf4fb-9bdc-40e3-942b-8dcaffe7fa11 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:41.084261 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-384cf4fb-9bdc-40e3-942b-8dcaffe7fa11 service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:41.117082 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1425/2844] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:42:40 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 412 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:42:42.047854 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abf11fb4-e996-4bc3-84a4-a5c1629663e3 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:42.048458 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abf11fb4-e996-4bc3-84a4-a5c1629663e3 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:42.048914 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abf11fb4-e996-4bc3-84a4-a5c1629663e3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:42.058948 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:42:42.090679 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-abf11fb4-e996-4bc3-84a4-a5c1629663e3 None None] MFA Rules not processed for user `f20085430c7e4d5983c4880fe8bc2b58`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:42:42.092205 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-abf11fb4-e996-4bc3-84a4-a5c1629663e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:42.186802 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1420/2845] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:42:42 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:42:42.219083 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-384cf4fb-9bdc-40e3-942b-8dcaffe7fa11 service cinder] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:42.221717 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-384cf4fb-9bdc-40e3-942b-8dcaffe7fa11 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:42.310100 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:42:42.355444 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-384cf4fb-9bdc-40e3-942b-8dcaffe7fa11 service cinder] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:42.362563 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7b213eb-421f-4e77-a268-f3e843cad492 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a35b5b7dea7b421da056f063d2436ffb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:42.363169 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7b213eb-421f-4e77-a268-f3e843cad492 service nova] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:42.363734 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7b213eb-421f-4e77-a268-f3e843cad492 service nova] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:42.364083 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7b213eb-421f-4e77-a268-f3e843cad492 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:42.439044 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7b213eb-421f-4e77-a268-f3e843cad492 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:42.440861 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7b213eb-421f-4e77-a268-f3e843cad492 service nova] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:42.452218 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1426/2846] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:42:42 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 234 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:42:43.873376 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abf11fb4-e996-4bc3-84a4-a5c1629663e3 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:43.964202 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abf11fb4-e996-4bc3-84a4-a5c1629663e3 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:43.970045 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-339cd61b-b3ce-480f-837f-3abe6cf82944 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:43.970687 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-339cd61b-b3ce-480f-837f-3abe6cf82944 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:43.971192 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-339cd61b-b3ce-480f-837f-3abe6cf82944 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:43.971820 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-339cd61b-b3ce-480f-837f-3abe6cf82944 admin admin] PATH_INFO: `/v3/users/d4da2a64dabc44c7aae3fd96fcfe0d12` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:43.973316 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-339cd61b-b3ce-480f-837f-3abe6cf82944 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d4da2a64dabc44c7aae3fd96fcfe0d12)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:43.975646 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-339cd61b-b3ce-480f-837f-3abe6cf82944 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:44.039121 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-339cd61b-b3ce-480f-837f-3abe6cf82944 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd4da2a64dabc44c7aae3fd96fcfe0d12'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:44.050028 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-339cd61b-b3ce-480f-837f-3abe6cf82944 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd4da2a64dabc44c7aae3fd96fcfe0d12'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:44.069280 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-339cd61b-b3ce-480f-837f-3abe6cf82944 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd4da2a64dabc44c7aae3fd96fcfe0d12'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:44.070023 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-339cd61b-b3ce-480f-837f-3abe6cf82944 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd4da2a64dabc44c7aae3fd96fcfe0d12'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:44.071420 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-339cd61b-b3ce-480f-837f-3abe6cf82944 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd4da2a64dabc44c7aae3fd96fcfe0d12'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:44.101142 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1421/2847] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:42:43 2022] DELETE /identity/v3/users/d4da2a64dabc44c7aae3fd96fcfe0d12 => generated 0 bytes in 229 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:42:44.106650 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bbcff2b-d417-416a-a426-1bea233efa9b None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:44.107780 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bbcff2b-d417-416a-a426-1bea233efa9b None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:44.108090 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bbcff2b-d417-416a-a426-1bea233efa9b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:44.137462 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-5bbcff2b-d417-416a-a426-1bea233efa9b None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:42:44.139091 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-5bbcff2b-d417-416a-a426-1bea233efa9b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:44.250914 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1427/2848] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:42:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 146 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 21:42:44.267290 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-339cd61b-b3ce-480f-837f-3abe6cf82944 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:44.270373 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-339cd61b-b3ce-480f-837f-3abe6cf82944 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:44.294211 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:42:44.458189 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-339cd61b-b3ce-480f-837f-3abe6cf82944 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:44.463961 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33812e41-0ebe-4d41-ab4b-376ef4ddd84d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:44.464642 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33812e41-0ebe-4d41-ab4b-376ef4ddd84d service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:44.464948 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33812e41-0ebe-4d41-ab4b-376ef4ddd84d service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:44.465419 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33812e41-0ebe-4d41-ab4b-376ef4ddd84d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:44.478700 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33812e41-0ebe-4d41-ab4b-376ef4ddd84d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:44.481011 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33812e41-0ebe-4d41-ab4b-376ef4ddd84d service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:44.503988 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1422/2849] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:42:44 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 238 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:42:44.877395 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bbcff2b-d417-416a-a426-1bea233efa9b None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:44.879174 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-5bbcff2b-d417-416a-a426-1bea233efa9b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:44.913541 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bbcff2b-d417-416a-a426-1bea233efa9b None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:44.919112 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88770542-3bec-4789-98ad-68b4378d5616 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:44.920686 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88770542-3bec-4789-98ad-68b4378d5616 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:44.920839 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88770542-3bec-4789-98ad-68b4378d5616 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:44.921343 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88770542-3bec-4789-98ad-68b4378d5616 admin admin] PATH_INFO: `/v3/projects/be24cb3c41b34759bfaa60a1f0cead52` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:44.923688 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88770542-3bec-4789-98ad-68b4378d5616 admin admin] RBAC: Authorizing `identity:delete_project(project_id=be24cb3c41b34759bfaa60a1f0cead52)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:44.925636 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88770542-3bec-4789-98ad-68b4378d5616 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:45.002724 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-88770542-3bec-4789-98ad-68b4378d5616 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'be24cb3c41b34759bfaa60a1f0cead52', 'request_id': 'req-88770542-3bec-4789-98ad-68b4378d5616'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:45.010296 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-88770542-3bec-4789-98ad-68b4378d5616 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'be24cb3c41b34759bfaa60a1f0cead52', 'request_id': 'req-88770542-3bec-4789-98ad-68b4378d5616'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:45.019003 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-88770542-3bec-4789-98ad-68b4378d5616 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'be24cb3c41b34759bfaa60a1f0cead52', 'request_id': 'req-88770542-3bec-4789-98ad-68b4378d5616'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:45.028424 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:42:45.036303 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-88770542-3bec-4789-98ad-68b4378d5616 admin admin] The token cache is being invalidate because project be24cb3c41b34759bfaa60a1f0cead52 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:42:45.036795 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-88770542-3bec-4789-98ad-68b4378d5616 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:45.037922 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-88770542-3bec-4789-98ad-68b4378d5616 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:45.041012 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1428/2850] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:42:44 2022] DELETE /identity/v3/projects/be24cb3c41b34759bfaa60a1f0cead52 => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:42:45.046089 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33812e41-0ebe-4d41-ab4b-376ef4ddd84d service neutron] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:45.056356 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33812e41-0ebe-4d41-ab4b-376ef4ddd84d service neutron] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:45.140370 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b81734a-8226-4137-b85d-4dc8dd70f05a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:45.141580 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b81734a-8226-4137-b85d-4dc8dd70f05a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:45.142165 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b81734a-8226-4137-b85d-4dc8dd70f05a admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:45.142469 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b81734a-8226-4137-b85d-4dc8dd70f05a admin admin] PATH_INFO: `/v3/users/de7588262bee4141bf2f1de2873664db` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:45.144174 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b81734a-8226-4137-b85d-4dc8dd70f05a admin admin] RBAC: Authorizing `identity:delete_user(user_id=de7588262bee4141bf2f1de2873664db)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:45.146942 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b81734a-8226-4137-b85d-4dc8dd70f05a admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:45.202606 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-5b81734a-8226-4137-b85d-4dc8dd70f05a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'de7588262bee4141bf2f1de2873664db'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:45.218861 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-5b81734a-8226-4137-b85d-4dc8dd70f05a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'de7588262bee4141bf2f1de2873664db'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:45.229535 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-5b81734a-8226-4137-b85d-4dc8dd70f05a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'de7588262bee4141bf2f1de2873664db'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:45.230295 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-5b81734a-8226-4137-b85d-4dc8dd70f05a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'de7588262bee4141bf2f1de2873664db'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:45.230963 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-5b81734a-8226-4137-b85d-4dc8dd70f05a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'de7588262bee4141bf2f1de2873664db'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:45.256230 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1423/2851] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:42:45 2022] DELETE /identity/v3/users/de7588262bee4141bf2f1de2873664db => generated 0 bytes in 211 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:42:45.682175 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88770542-3bec-4789-98ad-68b4378d5616 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:45.684427 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-88770542-3bec-4789-98ad-68b4378d5616 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:45.704881 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88770542-3bec-4789-98ad-68b4378d5616 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:45.785025 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b68125ab-bd1e-47e2-b3f2-48bb8fdd11d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:45.785693 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b68125ab-bd1e-47e2-b3f2-48bb8fdd11d0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:45.786003 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b68125ab-bd1e-47e2-b3f2-48bb8fdd11d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:45.786472 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b68125ab-bd1e-47e2-b3f2-48bb8fdd11d0 admin admin] PATH_INFO: `/v3/projects/38925e4eaa5041a5882f8c6166dff34e` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:45.787678 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b68125ab-bd1e-47e2-b3f2-48bb8fdd11d0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=38925e4eaa5041a5882f8c6166dff34e)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:45.789640 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b68125ab-bd1e-47e2-b3f2-48bb8fdd11d0 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:45.855106 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-b68125ab-bd1e-47e2-b3f2-48bb8fdd11d0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '38925e4eaa5041a5882f8c6166dff34e', 'request_id': 'req-b68125ab-bd1e-47e2-b3f2-48bb8fdd11d0'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:45.865471 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-b68125ab-bd1e-47e2-b3f2-48bb8fdd11d0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '38925e4eaa5041a5882f8c6166dff34e', 'request_id': 'req-b68125ab-bd1e-47e2-b3f2-48bb8fdd11d0'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:45.874369 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-b68125ab-bd1e-47e2-b3f2-48bb8fdd11d0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '38925e4eaa5041a5882f8c6166dff34e', 'request_id': 'req-b68125ab-bd1e-47e2-b3f2-48bb8fdd11d0'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:45.893926 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-b68125ab-bd1e-47e2-b3f2-48bb8fdd11d0 admin admin] The token cache is being invalidate because project 38925e4eaa5041a5882f8c6166dff34e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:42:45.894788 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-b68125ab-bd1e-47e2-b3f2-48bb8fdd11d0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:45.896208 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-b68125ab-bd1e-47e2-b3f2-48bb8fdd11d0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:45.898051 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1429/2852] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:42:45 2022] DELETE /identity/v3/projects/38925e4eaa5041a5882f8c6166dff34e => generated 0 bytes in 217 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:42:45.902268 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b81734a-8226-4137-b85d-4dc8dd70f05a admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:45.906011 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-5b81734a-8226-4137-b85d-4dc8dd70f05a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:45.928995 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b81734a-8226-4137-b85d-4dc8dd70f05a admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:45.962254 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:42:46.006622 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffe51c98-0e73-4920-9777-b1ad4e0c6591 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:46.007325 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffe51c98-0e73-4920-9777-b1ad4e0c6591 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:46.007638 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffe51c98-0e73-4920-9777-b1ad4e0c6591 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:46.007971 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffe51c98-0e73-4920-9777-b1ad4e0c6591 admin admin] PATH_INFO: `/v3/users/9da169756d1f45ee89cc6e0fad66d34e` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:46.009138 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffe51c98-0e73-4920-9777-b1ad4e0c6591 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9da169756d1f45ee89cc6e0fad66d34e)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:46.011275 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffe51c98-0e73-4920-9777-b1ad4e0c6591 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:46.091248 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-ffe51c98-0e73-4920-9777-b1ad4e0c6591 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9da169756d1f45ee89cc6e0fad66d34e'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:46.099095 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-ffe51c98-0e73-4920-9777-b1ad4e0c6591 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9da169756d1f45ee89cc6e0fad66d34e'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:46.107619 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-ffe51c98-0e73-4920-9777-b1ad4e0c6591 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9da169756d1f45ee89cc6e0fad66d34e'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:46.108383 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-ffe51c98-0e73-4920-9777-b1ad4e0c6591 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9da169756d1f45ee89cc6e0fad66d34e'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:46.109133 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-ffe51c98-0e73-4920-9777-b1ad4e0c6591 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9da169756d1f45ee89cc6e0fad66d34e'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:46.129079 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1424/2853] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:42:45 2022] DELETE /identity/v3/users/9da169756d1f45ee89cc6e0fad66d34e => generated 0 bytes in 227 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:42:46.449481 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b68125ab-bd1e-47e2-b3f2-48bb8fdd11d0 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:46.452989 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-b68125ab-bd1e-47e2-b3f2-48bb8fdd11d0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:46.474857 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b68125ab-bd1e-47e2-b3f2-48bb8fdd11d0 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:46.557330 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcf8f551-412b-4c6c-878b-6e226ac78e65 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:46.558020 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcf8f551-412b-4c6c-878b-6e226ac78e65 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:46.558301 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcf8f551-412b-4c6c-878b-6e226ac78e65 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:46.558580 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcf8f551-412b-4c6c-878b-6e226ac78e65 admin admin] PATH_INFO: `/v3/projects/d34f3d31b81f4864a868e7eb0f83f736` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:46.559754 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcf8f551-412b-4c6c-878b-6e226ac78e65 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d34f3d31b81f4864a868e7eb0f83f736)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:46.562821 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcf8f551-412b-4c6c-878b-6e226ac78e65 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:46.616608 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:42:46.654028 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-dcf8f551-412b-4c6c-878b-6e226ac78e65 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd34f3d31b81f4864a868e7eb0f83f736', 'request_id': 'req-dcf8f551-412b-4c6c-878b-6e226ac78e65'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:46.662235 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-dcf8f551-412b-4c6c-878b-6e226ac78e65 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd34f3d31b81f4864a868e7eb0f83f736', 'request_id': 'req-dcf8f551-412b-4c6c-878b-6e226ac78e65'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:46.680867 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-dcf8f551-412b-4c6c-878b-6e226ac78e65 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd34f3d31b81f4864a868e7eb0f83f736', 'request_id': 'req-dcf8f551-412b-4c6c-878b-6e226ac78e65'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:46.699724 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-dcf8f551-412b-4c6c-878b-6e226ac78e65 admin admin] The token cache is being invalidate because project d34f3d31b81f4864a868e7eb0f83f736 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:42:46.700891 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-dcf8f551-412b-4c6c-878b-6e226ac78e65 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:46.702569 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-dcf8f551-412b-4c6c-878b-6e226ac78e65 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:46.704398 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1430/2854] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:42:46 2022] DELETE /identity/v3/projects/d34f3d31b81f4864a868e7eb0f83f736 => generated 0 bytes in 256 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:42:46.713101 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6d63011-727c-4198-aef8-334c1efb7f72 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:46.713384 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6d63011-727c-4198-aef8-334c1efb7f72 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:46.713660 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6d63011-727c-4198-aef8-334c1efb7f72 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:46.744069 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-c6d63011-727c-4198-aef8-334c1efb7f72 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:42:46.745234 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-c6d63011-727c-4198-aef8-334c1efb7f72 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:46.840222 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1425/2855] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:42:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:42:46.843416 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7142a167-ac40-41d3-b01e-9b18933dcd65 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:46.843709 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7142a167-ac40-41d3-b01e-9b18933dcd65 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:46.844025 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7142a167-ac40-41d3-b01e-9b18933dcd65 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:46.869350 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-7142a167-ac40-41d3-b01e-9b18933dcd65 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:42:46.870878 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-7142a167-ac40-41d3-b01e-9b18933dcd65 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:46.880906 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1431/2856] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:42:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:42:46.885162 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6d63011-727c-4198-aef8-334c1efb7f72 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:46.897634 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6d63011-727c-4198-aef8-334c1efb7f72 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:46.907477 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d6abd1d-df39-4b03-b7fd-44532760fd63 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:46.908200 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d6abd1d-df39-4b03-b7fd-44532760fd63 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:46.908568 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d6abd1d-df39-4b03-b7fd-44532760fd63 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:46.908910 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d6abd1d-df39-4b03-b7fd-44532760fd63 admin admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:46.909966 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d6abd1d-df39-4b03-b7fd-44532760fd63 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:42:46.910088 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d6abd1d-df39-4b03-b7fd-44532760fd63 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:46.911759 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d6abd1d-df39-4b03-b7fd-44532760fd63 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:46.926136 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1426/2857] 10.222.0.90 () {58 vars in 1153 bytes} [Mon Aug 29 21:42:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:42:46.930770 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7142a167-ac40-41d3-b01e-9b18933dcd65 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:46.941402 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7142a167-ac40-41d3-b01e-9b18933dcd65 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:46.955815 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdb4f0e2-3234-4e5d-942f-30a10db42b2e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:46.957008 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdb4f0e2-3234-4e5d-942f-30a10db42b2e admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:46.957557 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdb4f0e2-3234-4e5d-942f-30a10db42b2e admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:46.958092 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdb4f0e2-3234-4e5d-942f-30a10db42b2e admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:46.959168 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdb4f0e2-3234-4e5d-942f-30a10db42b2e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:46.962533 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdb4f0e2-3234-4e5d-942f-30a10db42b2e admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:47.000745 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1432/2858] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:42:46 2022] POST /identity/v3/projects => generated 359 bytes in 71 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 21:42:47.005125 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d6abd1d-df39-4b03-b7fd-44532760fd63 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:47.016721 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d6abd1d-df39-4b03-b7fd-44532760fd63 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:47.054718 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:42:47.087253 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f3505b3-5045-48bb-b5fb-4823204ca5c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:47.088032 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f3505b3-5045-48bb-b5fb-4823204ca5c0 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:47.088321 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f3505b3-5045-48bb-b5fb-4823204ca5c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:47.088628 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f3505b3-5045-48bb-b5fb-4823204ca5c0 admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:47.089342 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f3505b3-5045-48bb-b5fb-4823204ca5c0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:47.091358 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f3505b3-5045-48bb-b5fb-4823204ca5c0 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:47.120647 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1427/2859] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:42:47 2022] POST /identity/v3/users => generated 280 bytes in 116 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:42:47.125873 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdb4f0e2-3234-4e5d-942f-30a10db42b2e admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:47.138564 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdb4f0e2-3234-4e5d-942f-30a10db42b2e admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:47.143682 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0297b09-3565-4e83-9009-1568b94175e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:47.144376 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0297b09-3565-4e83-9009-1568b94175e9 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:47.144726 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0297b09-3565-4e83-9009-1568b94175e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:47.145060 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0297b09-3565-4e83-9009-1568b94175e9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:47.146068 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0297b09-3565-4e83-9009-1568b94175e9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:47.147718 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0297b09-3565-4e83-9009-1568b94175e9 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:47.158975 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1433/2860] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:42:47 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:42:47.163139 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f3505b3-5045-48bb-b5fb-4823204ca5c0 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:47.176301 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f3505b3-5045-48bb-b5fb-4823204ca5c0 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:47.182493 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-957f0dcc-6f68-4fdb-9bb3-906b7aaa7910 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:47.183487 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-957f0dcc-6f68-4fdb-9bb3-906b7aaa7910 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:47.183898 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-957f0dcc-6f68-4fdb-9bb3-906b7aaa7910 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:47.184321 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-957f0dcc-6f68-4fdb-9bb3-906b7aaa7910 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:47.186144 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-957f0dcc-6f68-4fdb-9bb3-906b7aaa7910 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:47.189377 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-957f0dcc-6f68-4fdb-9bb3-906b7aaa7910 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:47.205165 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1428/2861] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:42:47 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:42:47.210205 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0297b09-3565-4e83-9009-1568b94175e9 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:47.221879 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0297b09-3565-4e83-9009-1568b94175e9 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:47.226857 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdf0fc3e-ebfc-4a88-80d8-9f3374f608e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:47.227701 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf0fc3e-ebfc-4a88-80d8-9f3374f608e4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:47.228092 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf0fc3e-ebfc-4a88-80d8-9f3374f608e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:47.228458 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf0fc3e-ebfc-4a88-80d8-9f3374f608e4 admin admin] PATH_INFO: `/v3/projects/698fb0620554438996746bc8a1790cfb/users/e3b96b67736d43b2921c67b4aafe3b45/roles/02e242d5520740d4b77a5f4008c7eb57` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:47.246598 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdf0fc3e-ebfc-4a88-80d8-9f3374f608e4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=698fb0620554438996746bc8a1790cfb, user_id=e3b96b67736d43b2921c67b4aafe3b45, role_id=02e242d5520740d4b77a5f4008c7eb57)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:47.252275 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdf0fc3e-ebfc-4a88-80d8-9f3374f608e4 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:47.270901 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1434/2862] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:42:47 2022] PUT /identity/v3/projects/698fb0620554438996746bc8a1790cfb/users/e3b96b67736d43b2921c67b4aafe3b45/roles/02e242d5520740d4b77a5f4008c7eb57 => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 21:42:47.282298 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-736470cd-9ad0-4fc7-807a-f47741e252df None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:47.282894 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-736470cd-9ad0-4fc7-807a-f47741e252df None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:47.283491 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-736470cd-9ad0-4fc7-807a-f47741e252df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:47.326208 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-736470cd-9ad0-4fc7-807a-f47741e252df None None] MFA Rules not processed for user `e3b96b67736d43b2921c67b4aafe3b45`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:42:47.327671 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-736470cd-9ad0-4fc7-807a-f47741e252df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:47.397234 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1429/2863] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:42:47 2022] POST /identity/v3/auth/tokens => generated 3236 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:42:47.433757 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdf0fc3e-ebfc-4a88-80d8-9f3374f608e4 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:47.436985 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-cdf0fc3e-ebfc-4a88-80d8-9f3374f608e4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:47.504904 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdf0fc3e-ebfc-4a88-80d8-9f3374f608e4 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:47.509282 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03926a92-0734-4bf9-bc21-db7bbac4dbf6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dc226f7ddcb49ecacb51ce57fdfc11b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:47.510087 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03926a92-0734-4bf9-bc21-db7bbac4dbf6 service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:47.510480 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03926a92-0734-4bf9-bc21-db7bbac4dbf6 service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:47.510835 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03926a92-0734-4bf9-bc21-db7bbac4dbf6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:47.522413 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03926a92-0734-4bf9-bc21-db7bbac4dbf6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:47.524331 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03926a92-0734-4bf9-bc21-db7bbac4dbf6 service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:47.538983 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1435/2864] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:42:47 2022] GET /identity/v3/auth/tokens => generated 3236 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:42:47.556371 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-736470cd-9ad0-4fc7-807a-f47741e252df None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:47.636595 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-736470cd-9ad0-4fc7-807a-f47741e252df None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:47.642524 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38990e1c-fe20-48b3-ac51-1307c3313558 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:47.642990 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38990e1c-fe20-48b3-ac51-1307c3313558 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:47.643277 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38990e1c-fe20-48b3-ac51-1307c3313558 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:47.643552 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38990e1c-fe20-48b3-ac51-1307c3313558 admin admin] PATH_INFO: `/v3/users/e3b96b67736d43b2921c67b4aafe3b45` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:47.644750 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38990e1c-fe20-48b3-ac51-1307c3313558 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e3b96b67736d43b2921c67b4aafe3b45)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:47.646838 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38990e1c-fe20-48b3-ac51-1307c3313558 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:47.699518 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=90,keystone:INSERT=3,keystone:DELETE=7 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:42:47.701687 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-38990e1c-fe20-48b3-ac51-1307c3313558 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e3b96b67736d43b2921c67b4aafe3b45'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:47.711216 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-38990e1c-fe20-48b3-ac51-1307c3313558 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e3b96b67736d43b2921c67b4aafe3b45'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:47.719839 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-38990e1c-fe20-48b3-ac51-1307c3313558 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e3b96b67736d43b2921c67b4aafe3b45'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:47.720746 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-38990e1c-fe20-48b3-ac51-1307c3313558 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e3b96b67736d43b2921c67b4aafe3b45'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:47.721438 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-38990e1c-fe20-48b3-ac51-1307c3313558 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e3b96b67736d43b2921c67b4aafe3b45'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:47.755339 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1430/2865] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:42:47 2022] DELETE /identity/v3/users/e3b96b67736d43b2921c67b4aafe3b45 => generated 0 bytes in 199 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:42:47.759850 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b93c231-33d7-4702-8814-82562bd11f7a None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:47.760342 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b93c231-33d7-4702-8814-82562bd11f7a None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:47.760457 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b93c231-33d7-4702-8814-82562bd11f7a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:47.783867 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-6b93c231-33d7-4702-8814-82562bd11f7a None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:42:47.785254 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-6b93c231-33d7-4702-8814-82562bd11f7a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:47.839720 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=92,keystone:DELETE=5,keystone:INSERT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:42:47.850493 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1436/2866] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:42:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:42:47.864620 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38990e1c-fe20-48b3-ac51-1307c3313558 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:47.866084 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-38990e1c-fe20-48b3-ac51-1307c3313558 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:47.924757 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38990e1c-fe20-48b3-ac51-1307c3313558 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:47.929019 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c382d559-4bfb-4af7-b0ef-72ff0b7cb927 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:47.929645 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c382d559-4bfb-4af7-b0ef-72ff0b7cb927 service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:47.930045 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c382d559-4bfb-4af7-b0ef-72ff0b7cb927 service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:47.930325 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c382d559-4bfb-4af7-b0ef-72ff0b7cb927 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:47.941419 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c382d559-4bfb-4af7-b0ef-72ff0b7cb927 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:47.942981 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c382d559-4bfb-4af7-b0ef-72ff0b7cb927 service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:47.954267 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1431/2867] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:42:47 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:42:48.246475 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b93c231-33d7-4702-8814-82562bd11f7a None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:48.248560 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-6b93c231-33d7-4702-8814-82562bd11f7a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:48.261251 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b93c231-33d7-4702-8814-82562bd11f7a None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:48.266856 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-620f8ad3-ce0f-475e-890f-f00c2cc313aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:48.267593 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-620f8ad3-ce0f-475e-890f-f00c2cc313aa admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:48.267972 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-620f8ad3-ce0f-475e-890f-f00c2cc313aa admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:48.268329 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-620f8ad3-ce0f-475e-890f-f00c2cc313aa admin admin] PATH_INFO: `/v3/projects/698fb0620554438996746bc8a1790cfb` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:48.269674 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-620f8ad3-ce0f-475e-890f-f00c2cc313aa admin admin] RBAC: Authorizing `identity:delete_project(project_id=698fb0620554438996746bc8a1790cfb)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:48.271622 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-620f8ad3-ce0f-475e-890f-f00c2cc313aa admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:48.322558 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-620f8ad3-ce0f-475e-890f-f00c2cc313aa admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '698fb0620554438996746bc8a1790cfb', 'request_id': 'req-620f8ad3-ce0f-475e-890f-f00c2cc313aa'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:48.330238 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-620f8ad3-ce0f-475e-890f-f00c2cc313aa admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '698fb0620554438996746bc8a1790cfb', 'request_id': 'req-620f8ad3-ce0f-475e-890f-f00c2cc313aa'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:48.335872 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-620f8ad3-ce0f-475e-890f-f00c2cc313aa admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '698fb0620554438996746bc8a1790cfb', 'request_id': 'req-620f8ad3-ce0f-475e-890f-f00c2cc313aa'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:48.351458 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-620f8ad3-ce0f-475e-890f-f00c2cc313aa admin admin] The token cache is being invalidate because project 698fb0620554438996746bc8a1790cfb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:42:48.351458 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-620f8ad3-ce0f-475e-890f-f00c2cc313aa admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:48.351458 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-620f8ad3-ce0f-475e-890f-f00c2cc313aa admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:48.355188 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1437/2868] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:42:48 2022] DELETE /identity/v3/projects/698fb0620554438996746bc8a1790cfb => generated 0 bytes in 106 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:42:48.357678 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e64ef0b-0eab-492a-9864-716713a88e2b None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:48.358187 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e64ef0b-0eab-492a-9864-716713a88e2b None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:48.358304 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e64ef0b-0eab-492a-9864-716713a88e2b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:48.379338 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-4e64ef0b-0eab-492a-9864-716713a88e2b None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:42:48.380699 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-4e64ef0b-0eab-492a-9864-716713a88e2b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:48.448399 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1432/2869] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:42:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:42:48.458027 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9f29e7d-3ddb-4fbf-a7f5-b62bb061e184 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:48.458436 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9f29e7d-3ddb-4fbf-a7f5-b62bb061e184 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:48.458786 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9f29e7d-3ddb-4fbf-a7f5-b62bb061e184 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:48.480385 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-d9f29e7d-3ddb-4fbf-a7f5-b62bb061e184 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:42:48.481927 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-d9f29e7d-3ddb-4fbf-a7f5-b62bb061e184 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:48.509570 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1438/2870] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:42:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 52 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:42:48.514199 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e64ef0b-0eab-492a-9864-716713a88e2b None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:48.523956 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e64ef0b-0eab-492a-9864-716713a88e2b None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:48.529728 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50b0e5b7-65cc-44cc-bb9b-700c4a4c06c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:48.530377 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50b0e5b7-65cc-44cc-bb9b-700c4a4c06c3 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:48.530651 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50b0e5b7-65cc-44cc-bb9b-700c4a4c06c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:48.530919 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50b0e5b7-65cc-44cc-bb9b-700c4a4c06c3 admin admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:48.531570 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50b0e5b7-65cc-44cc-bb9b-700c4a4c06c3 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:42:48.531873 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50b0e5b7-65cc-44cc-bb9b-700c4a4c06c3 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:48.533546 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50b0e5b7-65cc-44cc-bb9b-700c4a4c06c3 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:48.546489 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1433/2871] 10.222.0.90 () {58 vars in 1153 bytes} [Mon Aug 29 21:42:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 21:42:48.550874 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9f29e7d-3ddb-4fbf-a7f5-b62bb061e184 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:48.561043 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9f29e7d-3ddb-4fbf-a7f5-b62bb061e184 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:48.566134 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7564f5bd-a9fc-48dd-ac11-32ec4e89ef75 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:48.566602 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7564f5bd-a9fc-48dd-ac11-32ec4e89ef75 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:48.566885 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7564f5bd-a9fc-48dd-ac11-32ec4e89ef75 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:48.567162 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7564f5bd-a9fc-48dd-ac11-32ec4e89ef75 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:48.567819 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7564f5bd-a9fc-48dd-ac11-32ec4e89ef75 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:48.569778 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7564f5bd-a9fc-48dd-ac11-32ec4e89ef75 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:48.602497 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1439/2872] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:42:48 2022] POST /identity/v3/projects => generated 375 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:42:48.606703 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50b0e5b7-65cc-44cc-bb9b-700c4a4c06c3 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:48.620304 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50b0e5b7-65cc-44cc-bb9b-700c4a4c06c3 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:48.691075 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=96,keystone:INSERT=1,keystone:DELETE=3 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:42:48.702658 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fb04c4c-25d9-4b63-9aca-f2f6b7d3f833 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:48.703494 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fb04c4c-25d9-4b63-9aca-f2f6b7d3f833 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:48.703979 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fb04c4c-25d9-4b63-9aca-f2f6b7d3f833 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:48.704411 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fb04c4c-25d9-4b63-9aca-f2f6b7d3f833 admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:48.705471 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fb04c4c-25d9-4b63-9aca-f2f6b7d3f833 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:48.707589 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fb04c4c-25d9-4b63-9aca-f2f6b7d3f833 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:48.730403 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1434/2873] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:42:48 2022] POST /identity/v3/users => generated 288 bytes in 124 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 21:42:48.735494 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7564f5bd-a9fc-48dd-ac11-32ec4e89ef75 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:48.746487 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7564f5bd-a9fc-48dd-ac11-32ec4e89ef75 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:48.752975 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-100b090a-c295-44a5-9b2f-09db0e5c7d25 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:48.754186 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-100b090a-c295-44a5-9b2f-09db0e5c7d25 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:48.754186 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-100b090a-c295-44a5-9b2f-09db0e5c7d25 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:48.754186 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-100b090a-c295-44a5-9b2f-09db0e5c7d25 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:48.754849 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-100b090a-c295-44a5-9b2f-09db0e5c7d25 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:48.756638 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-100b090a-c295-44a5-9b2f-09db0e5c7d25 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:48.770154 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1440/2874] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:42:48 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:42:48.771222 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fb04c4c-25d9-4b63-9aca-f2f6b7d3f833 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:48.782270 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fb04c4c-25d9-4b63-9aca-f2f6b7d3f833 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:48.787607 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6310da9-c18e-4137-8a48-68f93f615860 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:48.788240 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6310da9-c18e-4137-8a48-68f93f615860 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:48.788521 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6310da9-c18e-4137-8a48-68f93f615860 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:48.788797 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6310da9-c18e-4137-8a48-68f93f615860 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:48.789616 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6310da9-c18e-4137-8a48-68f93f615860 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:48.791301 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6310da9-c18e-4137-8a48-68f93f615860 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:48.814436 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1435/2875] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:42:48 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 29 21:42:48.819089 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-100b090a-c295-44a5-9b2f-09db0e5c7d25 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:48.836522 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-100b090a-c295-44a5-9b2f-09db0e5c7d25 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:48.843771 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37e37e49-81de-4bff-baac-3bf8ca5c769d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:48.844463 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37e37e49-81de-4bff-baac-3bf8ca5c769d admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:48.844746 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37e37e49-81de-4bff-baac-3bf8ca5c769d admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:48.845025 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37e37e49-81de-4bff-baac-3bf8ca5c769d admin admin] PATH_INFO: `/v3/projects/e67a41e6b6184dfa88057cbbaeadd92c/users/10a4090bda824bdfaa060e8711c1982b/roles/02e242d5520740d4b77a5f4008c7eb57` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:48.861873 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37e37e49-81de-4bff-baac-3bf8ca5c769d admin admin] RBAC: Authorizing `identity:create_grant(project_id=e67a41e6b6184dfa88057cbbaeadd92c, user_id=10a4090bda824bdfaa060e8711c1982b, role_id=02e242d5520740d4b77a5f4008c7eb57)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:48.864568 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37e37e49-81de-4bff-baac-3bf8ca5c769d admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:48.879012 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1441/2876] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:42:48 2022] PUT /identity/v3/projects/e67a41e6b6184dfa88057cbbaeadd92c/users/10a4090bda824bdfaa060e8711c1982b/roles/02e242d5520740d4b77a5f4008c7eb57 => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:42:48.889570 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-666f1635-c6cd-4444-a445-b3eaa3bee6f9 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:48.889905 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-666f1635-c6cd-4444-a445-b3eaa3bee6f9 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:48.890187 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-666f1635-c6cd-4444-a445-b3eaa3bee6f9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:48.918318 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-666f1635-c6cd-4444-a445-b3eaa3bee6f9 None None] MFA Rules not processed for user `10a4090bda824bdfaa060e8711c1982b`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:42:48.920193 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-666f1635-c6cd-4444-a445-b3eaa3bee6f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:48.994049 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1436/2877] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:42:48 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:42:49.008379 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37e37e49-81de-4bff-baac-3bf8ca5c769d admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:49.010114 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-37e37e49-81de-4bff-baac-3bf8ca5c769d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:49.071208 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37e37e49-81de-4bff-baac-3bf8ca5c769d admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:49.074974 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da65d58d-c614-4112-b802-854b7a21a403 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dc226f7ddcb49ecacb51ce57fdfc11b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:49.075909 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da65d58d-c614-4112-b802-854b7a21a403 service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:49.076189 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da65d58d-c614-4112-b802-854b7a21a403 service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:49.076463 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da65d58d-c614-4112-b802-854b7a21a403 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:49.085593 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da65d58d-c614-4112-b802-854b7a21a403 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:49.087062 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da65d58d-c614-4112-b802-854b7a21a403 service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:49.096620 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1442/2878] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:42:49 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:42:53.261856 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-666f1635-c6cd-4444-a445-b3eaa3bee6f9 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:53.324307 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-666f1635-c6cd-4444-a445-b3eaa3bee6f9 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:53.330058 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-515a4b59-f363-4000-a5e0-d5a8fadd5da6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:53.330691 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-515a4b59-f363-4000-a5e0-d5a8fadd5da6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:53.330974 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-515a4b59-f363-4000-a5e0-d5a8fadd5da6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:53.331249 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-515a4b59-f363-4000-a5e0-d5a8fadd5da6 admin admin] PATH_INFO: `/v3/users/10a4090bda824bdfaa060e8711c1982b` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:53.332372 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-515a4b59-f363-4000-a5e0-d5a8fadd5da6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=10a4090bda824bdfaa060e8711c1982b)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:53.334316 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-515a4b59-f363-4000-a5e0-d5a8fadd5da6 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:53.380781 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-515a4b59-f363-4000-a5e0-d5a8fadd5da6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '10a4090bda824bdfaa060e8711c1982b'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:53.392110 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-515a4b59-f363-4000-a5e0-d5a8fadd5da6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '10a4090bda824bdfaa060e8711c1982b'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:53.401346 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-515a4b59-f363-4000-a5e0-d5a8fadd5da6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '10a4090bda824bdfaa060e8711c1982b'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:53.402053 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-515a4b59-f363-4000-a5e0-d5a8fadd5da6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '10a4090bda824bdfaa060e8711c1982b'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:53.402664 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-515a4b59-f363-4000-a5e0-d5a8fadd5da6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '10a4090bda824bdfaa060e8711c1982b'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:53.404077 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=88,keystone:INSERT=4,keystone:DELETE=8 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:42:53.425561 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1437/2879] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:42:53 2022] DELETE /identity/v3/users/10a4090bda824bdfaa060e8711c1982b => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:42:53.430233 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4a1d365-26f2-444d-8552-c4f332efeea8 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:53.430774 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4a1d365-26f2-444d-8552-c4f332efeea8 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:53.431088 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4a1d365-26f2-444d-8552-c4f332efeea8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:53.454700 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-c4a1d365-26f2-444d-8552-c4f332efeea8 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:42:53.455945 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-c4a1d365-26f2-444d-8552-c4f332efeea8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:53.469518 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:42:53.523007 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1443/2880] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:42:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:42:53.531105 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-515a4b59-f363-4000-a5e0-d5a8fadd5da6 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:53.532571 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-515a4b59-f363-4000-a5e0-d5a8fadd5da6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:53.591710 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-515a4b59-f363-4000-a5e0-d5a8fadd5da6 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:53.595942 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb9fedb7-cb79-4e15-af16-18e235087c79 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:53.596548 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb9fedb7-cb79-4e15-af16-18e235087c79 service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:53.596833 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb9fedb7-cb79-4e15-af16-18e235087c79 service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:53.597103 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb9fedb7-cb79-4e15-af16-18e235087c79 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:53.606234 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb9fedb7-cb79-4e15-af16-18e235087c79 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:53.607687 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb9fedb7-cb79-4e15-af16-18e235087c79 service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:53.618364 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1438/2881] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:42:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:42:53.939769 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4a1d365-26f2-444d-8552-c4f332efeea8 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:53.941827 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-c4a1d365-26f2-444d-8552-c4f332efeea8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:53.970335 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4a1d365-26f2-444d-8552-c4f332efeea8 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:53.978638 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6af35aaa-5ff9-43d3-894f-abb97185bcf3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:53.979444 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6af35aaa-5ff9-43d3-894f-abb97185bcf3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:53.979666 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6af35aaa-5ff9-43d3-894f-abb97185bcf3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:53.980099 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6af35aaa-5ff9-43d3-894f-abb97185bcf3 admin admin] PATH_INFO: `/v3/projects/e67a41e6b6184dfa88057cbbaeadd92c` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:53.981396 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6af35aaa-5ff9-43d3-894f-abb97185bcf3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e67a41e6b6184dfa88057cbbaeadd92c)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:53.984009 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6af35aaa-5ff9-43d3-894f-abb97185bcf3 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:54.033424 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-6af35aaa-5ff9-43d3-894f-abb97185bcf3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e67a41e6b6184dfa88057cbbaeadd92c', 'request_id': 'req-6af35aaa-5ff9-43d3-894f-abb97185bcf3'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:54.039639 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-6af35aaa-5ff9-43d3-894f-abb97185bcf3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e67a41e6b6184dfa88057cbbaeadd92c', 'request_id': 'req-6af35aaa-5ff9-43d3-894f-abb97185bcf3'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:54.045695 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-6af35aaa-5ff9-43d3-894f-abb97185bcf3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e67a41e6b6184dfa88057cbbaeadd92c', 'request_id': 'req-6af35aaa-5ff9-43d3-894f-abb97185bcf3'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:54.062863 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-6af35aaa-5ff9-43d3-894f-abb97185bcf3 admin admin] The token cache is being invalidate because project e67a41e6b6184dfa88057cbbaeadd92c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:42:54.063145 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-6af35aaa-5ff9-43d3-894f-abb97185bcf3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:54.063930 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-6af35aaa-5ff9-43d3-894f-abb97185bcf3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:54.065684 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1444/2882] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:42:53 2022] DELETE /identity/v3/projects/e67a41e6b6184dfa88057cbbaeadd92c => generated 0 bytes in 130 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:42:54.070968 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd4d8064-5211-446d-81bf-52d2ef6be404 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:54.071282 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd4d8064-5211-446d-81bf-52d2ef6be404 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:54.071559 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd4d8064-5211-446d-81bf-52d2ef6be404 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:54.092605 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-cd4d8064-5211-446d-81bf-52d2ef6be404 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:42:54.093864 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-cd4d8064-5211-446d-81bf-52d2ef6be404 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:54.158860 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1439/2883] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:42:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:42:54.166667 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50f7ae45-acb3-4a1c-926b-ec3fc6e45264 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:54.166951 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50f7ae45-acb3-4a1c-926b-ec3fc6e45264 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:54.167223 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50f7ae45-acb3-4a1c-926b-ec3fc6e45264 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:54.188290 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-50f7ae45-acb3-4a1c-926b-ec3fc6e45264 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:42:54.189488 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-50f7ae45-acb3-4a1c-926b-ec3fc6e45264 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:54.197213 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1445/2884] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:42:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:42:54.201698 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd4d8064-5211-446d-81bf-52d2ef6be404 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:54.212176 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd4d8064-5211-446d-81bf-52d2ef6be404 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:54.216757 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f371539e-2229-4623-b5fd-1525f79df1a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:54.218241 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f371539e-2229-4623-b5fd-1525f79df1a1 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:54.218517 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f371539e-2229-4623-b5fd-1525f79df1a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:54.218785 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f371539e-2229-4623-b5fd-1525f79df1a1 admin admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:54.219429 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f371539e-2229-4623-b5fd-1525f79df1a1 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:42:54.219729 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f371539e-2229-4623-b5fd-1525f79df1a1 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:54.221307 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f371539e-2229-4623-b5fd-1525f79df1a1 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:54.234104 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1440/2885] 10.222.0.90 () {58 vars in 1153 bytes} [Mon Aug 29 21:42:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:42:54.238591 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50f7ae45-acb3-4a1c-926b-ec3fc6e45264 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:54.248451 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50f7ae45-acb3-4a1c-926b-ec3fc6e45264 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:54.253472 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5698b1c-7dab-4027-ac2b-796f5b37a57b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:54.254118 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5698b1c-7dab-4027-ac2b-796f5b37a57b admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:54.254398 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5698b1c-7dab-4027-ac2b-796f5b37a57b admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:54.254666 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5698b1c-7dab-4027-ac2b-796f5b37a57b admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:54.255308 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5698b1c-7dab-4027-ac2b-796f5b37a57b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:54.257196 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5698b1c-7dab-4027-ac2b-796f5b37a57b admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:54.292314 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1446/2886] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:42:54 2022] POST /identity/v3/projects => generated 389 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:42:54.297525 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f371539e-2229-4623-b5fd-1525f79df1a1 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:54.307742 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f371539e-2229-4623-b5fd-1525f79df1a1 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:54.364164 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f24fff05-163e-4154-96dc-637aacb421c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:54.364769 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f24fff05-163e-4154-96dc-637aacb421c7 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:54.365049 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f24fff05-163e-4154-96dc-637aacb421c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:54.365322 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f24fff05-163e-4154-96dc-637aacb421c7 admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:54.365996 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f24fff05-163e-4154-96dc-637aacb421c7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:54.367907 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f24fff05-163e-4154-96dc-637aacb421c7 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:54.383927 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=95,keystone:DELETE=2,keystone:INSERT=3 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:42:54.392587 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1441/2887] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:42:54 2022] POST /identity/v3/users => generated 295 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:42:54.393721 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5698b1c-7dab-4027-ac2b-796f5b37a57b admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:54.404032 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5698b1c-7dab-4027-ac2b-796f5b37a57b admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:54.409508 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-706309cd-70c1-4346-838c-1ea826113c69 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:54.410141 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-706309cd-70c1-4346-838c-1ea826113c69 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:54.410428 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-706309cd-70c1-4346-838c-1ea826113c69 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:54.410703 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-706309cd-70c1-4346-838c-1ea826113c69 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:54.411341 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-706309cd-70c1-4346-838c-1ea826113c69 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:54.412925 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-706309cd-70c1-4346-838c-1ea826113c69 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:54.423567 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1447/2888] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:42:54 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 29 21:42:54.428505 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f24fff05-163e-4154-96dc-637aacb421c7 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:54.442625 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f24fff05-163e-4154-96dc-637aacb421c7 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:54.447824 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26867550-17f7-40b7-9be4-3b3b661217ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:54.448483 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26867550-17f7-40b7-9be4-3b3b661217ee admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:54.448762 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26867550-17f7-40b7-9be4-3b3b661217ee admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:54.449038 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26867550-17f7-40b7-9be4-3b3b661217ee admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:54.449710 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26867550-17f7-40b7-9be4-3b3b661217ee admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:54.451594 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26867550-17f7-40b7-9be4-3b3b661217ee admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:54.463365 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1442/2889] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:42:54 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:42:54.468333 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-706309cd-70c1-4346-838c-1ea826113c69 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:54.478696 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-706309cd-70c1-4346-838c-1ea826113c69 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:54.484785 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c616f4c9-6e24-43ed-97b6-0819e504e048 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:54.485443 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c616f4c9-6e24-43ed-97b6-0819e504e048 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:54.485731 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c616f4c9-6e24-43ed-97b6-0819e504e048 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:54.486042 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c616f4c9-6e24-43ed-97b6-0819e504e048 admin admin] PATH_INFO: `/v3/projects/9267783151d64683997ea5302e2e6ec1/users/a82ca052e3bd4b898bf5623f4e1ed711/roles/02e242d5520740d4b77a5f4008c7eb57` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:54.501941 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c616f4c9-6e24-43ed-97b6-0819e504e048 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9267783151d64683997ea5302e2e6ec1, user_id=a82ca052e3bd4b898bf5623f4e1ed711, role_id=02e242d5520740d4b77a5f4008c7eb57)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:54.504593 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c616f4c9-6e24-43ed-97b6-0819e504e048 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:54.518071 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1448/2890] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:42:54 2022] PUT /identity/v3/projects/9267783151d64683997ea5302e2e6ec1/users/a82ca052e3bd4b898bf5623f4e1ed711/roles/02e242d5520740d4b77a5f4008c7eb57 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:42:54.530383 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77dc4976-6262-48c6-b405-946ce2505f98 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:54.530683 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77dc4976-6262-48c6-b405-946ce2505f98 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:54.531315 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77dc4976-6262-48c6-b405-946ce2505f98 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:54.559055 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-77dc4976-6262-48c6-b405-946ce2505f98 None None] MFA Rules not processed for user `a82ca052e3bd4b898bf5623f4e1ed711`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:42:54.560259 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-77dc4976-6262-48c6-b405-946ce2505f98 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:54.630304 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1443/2891] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:42:54 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:42:54.642039 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c616f4c9-6e24-43ed-97b6-0819e504e048 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:54.643731 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-c616f4c9-6e24-43ed-97b6-0819e504e048 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:54.688757 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:42:54.704877 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c616f4c9-6e24-43ed-97b6-0819e504e048 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:54.708760 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeb971e1-3488-4950-a7c3-6f4c2d46caf3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dc226f7ddcb49ecacb51ce57fdfc11b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:54.709362 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeb971e1-3488-4950-a7c3-6f4c2d46caf3 service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:54.709638 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeb971e1-3488-4950-a7c3-6f4c2d46caf3 service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:54.709950 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeb971e1-3488-4950-a7c3-6f4c2d46caf3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:54.719711 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aeb971e1-3488-4950-a7c3-6f4c2d46caf3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:54.720938 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aeb971e1-3488-4950-a7c3-6f4c2d46caf3 service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:54.729770 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1449/2892] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:42:54 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:42:56.187499 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77dc4976-6262-48c6-b405-946ce2505f98 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:56.188974 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-77dc4976-6262-48c6-b405-946ce2505f98 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:56.260821 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77dc4976-6262-48c6-b405-946ce2505f98 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:56.266260 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b65f3374-7a12-4263-a0a8-dbb861fa7766 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:56.266904 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b65f3374-7a12-4263-a0a8-dbb861fa7766 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:56.267187 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b65f3374-7a12-4263-a0a8-dbb861fa7766 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:56.267464 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b65f3374-7a12-4263-a0a8-dbb861fa7766 admin admin] PATH_INFO: `/v3/users/f1c6f94c93dd456ca2108516d67032ca` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:56.268609 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b65f3374-7a12-4263-a0a8-dbb861fa7766 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f1c6f94c93dd456ca2108516d67032ca)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:56.270551 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b65f3374-7a12-4263-a0a8-dbb861fa7766 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:56.318341 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-b65f3374-7a12-4263-a0a8-dbb861fa7766 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f1c6f94c93dd456ca2108516d67032ca'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:56.326014 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-b65f3374-7a12-4263-a0a8-dbb861fa7766 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f1c6f94c93dd456ca2108516d67032ca'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:56.336741 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-b65f3374-7a12-4263-a0a8-dbb861fa7766 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f1c6f94c93dd456ca2108516d67032ca'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:56.337400 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-b65f3374-7a12-4263-a0a8-dbb861fa7766 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f1c6f94c93dd456ca2108516d67032ca'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:56.338058 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-b65f3374-7a12-4263-a0a8-dbb861fa7766 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f1c6f94c93dd456ca2108516d67032ca'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:56.353733 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:42:56.361325 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1444/2893] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:42:56 2022] DELETE /identity/v3/users/f1c6f94c93dd456ca2108516d67032ca => generated 0 bytes in 174 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:42:56.367256 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce088407-49cc-4987-94d8-6020ba0470b8 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:56.367787 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce088407-49cc-4987-94d8-6020ba0470b8 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:56.367958 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce088407-49cc-4987-94d8-6020ba0470b8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:56.390394 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-ce088407-49cc-4987-94d8-6020ba0470b8 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:42:56.393030 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-ce088407-49cc-4987-94d8-6020ba0470b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:56.474609 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1450/2894] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:42:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 21:42:56.484690 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b65f3374-7a12-4263-a0a8-dbb861fa7766 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:56.486180 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-b65f3374-7a12-4263-a0a8-dbb861fa7766 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:56.545514 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b65f3374-7a12-4263-a0a8-dbb861fa7766 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:56.549536 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c539c087-e2f6-4eea-ba21-31e5f6b50ff0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:56.550178 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c539c087-e2f6-4eea-ba21-31e5f6b50ff0 service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:56.550463 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c539c087-e2f6-4eea-ba21-31e5f6b50ff0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:56.550737 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c539c087-e2f6-4eea-ba21-31e5f6b50ff0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:56.559572 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c539c087-e2f6-4eea-ba21-31e5f6b50ff0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:56.562063 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c539c087-e2f6-4eea-ba21-31e5f6b50ff0 service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:56.572461 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1445/2895] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:42:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:42:56.838599 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce088407-49cc-4987-94d8-6020ba0470b8 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:56.840143 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-ce088407-49cc-4987-94d8-6020ba0470b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:56.862492 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce088407-49cc-4987-94d8-6020ba0470b8 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:56.868403 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe378fc1-258b-4761-9c73-cc95f89bbd1a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:56.869111 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe378fc1-258b-4761-9c73-cc95f89bbd1a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:56.869321 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe378fc1-258b-4761-9c73-cc95f89bbd1a admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:56.869639 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe378fc1-258b-4761-9c73-cc95f89bbd1a admin admin] PATH_INFO: `/v3/projects/f3ae1ee970c7429a9b68402502e57919` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:56.870768 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe378fc1-258b-4761-9c73-cc95f89bbd1a admin admin] RBAC: Authorizing `identity:delete_project(project_id=f3ae1ee970c7429a9b68402502e57919)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:56.872687 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe378fc1-258b-4761-9c73-cc95f89bbd1a admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:56.921943 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-fe378fc1-258b-4761-9c73-cc95f89bbd1a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f3ae1ee970c7429a9b68402502e57919', 'request_id': 'req-fe378fc1-258b-4761-9c73-cc95f89bbd1a'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:56.927950 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-fe378fc1-258b-4761-9c73-cc95f89bbd1a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f3ae1ee970c7429a9b68402502e57919', 'request_id': 'req-fe378fc1-258b-4761-9c73-cc95f89bbd1a'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:56.933711 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-fe378fc1-258b-4761-9c73-cc95f89bbd1a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f3ae1ee970c7429a9b68402502e57919', 'request_id': 'req-fe378fc1-258b-4761-9c73-cc95f89bbd1a'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:56.946190 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-fe378fc1-258b-4761-9c73-cc95f89bbd1a admin admin] The token cache is being invalidate because project f3ae1ee970c7429a9b68402502e57919 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:42:56.946598 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-fe378fc1-258b-4761-9c73-cc95f89bbd1a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:56.947225 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-fe378fc1-258b-4761-9c73-cc95f89bbd1a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:42:56.948757 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1451/2896] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:42:56 2022] DELETE /identity/v3/projects/f3ae1ee970c7429a9b68402502e57919 => generated 0 bytes in 111 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:42:56.954455 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a57063f7-8043-4133-ab56-c0df2c44e545 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:56.954737 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a57063f7-8043-4133-ab56-c0df2c44e545 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:56.955009 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a57063f7-8043-4133-ab56-c0df2c44e545 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:56.974910 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-a57063f7-8043-4133-ab56-c0df2c44e545 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:42:56.976101 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-a57063f7-8043-4133-ab56-c0df2c44e545 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:57.034218 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1446/2897] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:42:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:42:57.042072 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6dcbaa4-e573-4c44-900b-46f5b5ee1281 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:57.042363 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6dcbaa4-e573-4c44-900b-46f5b5ee1281 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:57.042640 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6dcbaa4-e573-4c44-900b-46f5b5ee1281 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:57.063003 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-d6dcbaa4-e573-4c44-900b-46f5b5ee1281 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:42:57.064227 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-d6dcbaa4-e573-4c44-900b-46f5b5ee1281 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:57.072183 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1452/2898] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:42:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:42:57.077284 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a57063f7-8043-4133-ab56-c0df2c44e545 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:57.087307 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a57063f7-8043-4133-ab56-c0df2c44e545 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:57.092874 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64932967-b907-489f-b136-52ab52f27d25 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:57.093625 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64932967-b907-489f-b136-52ab52f27d25 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:57.093974 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64932967-b907-489f-b136-52ab52f27d25 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:57.094252 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64932967-b907-489f-b136-52ab52f27d25 admin admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:57.094913 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64932967-b907-489f-b136-52ab52f27d25 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 21:42:57.095249 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64932967-b907-489f-b136-52ab52f27d25 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:57.096849 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64932967-b907-489f-b136-52ab52f27d25 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:57.109579 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1447/2899] 10.222.0.90 () {58 vars in 1153 bytes} [Mon Aug 29 21:42:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 21:42:57.114034 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6dcbaa4-e573-4c44-900b-46f5b5ee1281 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:57.124957 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6dcbaa4-e573-4c44-900b-46f5b5ee1281 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:57.129709 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad0c2208-1441-4dc1-b8c0-3ca75ee16f19 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:57.130642 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad0c2208-1441-4dc1-b8c0-3ca75ee16f19 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:57.130878 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad0c2208-1441-4dc1-b8c0-3ca75ee16f19 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:57.130991 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad0c2208-1441-4dc1-b8c0-3ca75ee16f19 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:57.131604 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad0c2208-1441-4dc1-b8c0-3ca75ee16f19 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:57.133491 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad0c2208-1441-4dc1-b8c0-3ca75ee16f19 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:57.160730 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1453/2900] 10.222.0.90 () {60 vars in 1153 bytes} [Mon Aug 29 21:42:57 2022] POST /identity/v3/projects => generated 397 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:42:57.165015 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64932967-b907-489f-b136-52ab52f27d25 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:57.175825 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64932967-b907-489f-b136-52ab52f27d25 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:57.233857 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fa810c4-ae0e-4335-b8c4-cde482ed49e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:57.234649 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fa810c4-ae0e-4335-b8c4-cde482ed49e6 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:57.234969 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fa810c4-ae0e-4335-b8c4-cde482ed49e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:57.235248 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fa810c4-ae0e-4335-b8c4-cde482ed49e6 admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:57.235884 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fa810c4-ae0e-4335-b8c4-cde482ed49e6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:57.237829 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fa810c4-ae0e-4335-b8c4-cde482ed49e6 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:57.260375 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1448/2901] 10.222.0.90 () {60 vars in 1144 bytes} [Mon Aug 29 21:42:57 2022] POST /identity/v3/users => generated 299 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 21:42:57.264908 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad0c2208-1441-4dc1-b8c0-3ca75ee16f19 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:57.275095 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad0c2208-1441-4dc1-b8c0-3ca75ee16f19 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:57.280213 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4dd9b3f-42c1-47d1-82e0-29589c2170a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:57.280829 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4dd9b3f-42c1-47d1-82e0-29589c2170a0 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:57.281101 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4dd9b3f-42c1-47d1-82e0-29589c2170a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:57.281370 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4dd9b3f-42c1-47d1-82e0-29589c2170a0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:57.282029 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4dd9b3f-42c1-47d1-82e0-29589c2170a0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:57.283731 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4dd9b3f-42c1-47d1-82e0-29589c2170a0 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:57.294569 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1454/2902] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:42:57 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:42:57.299308 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fa810c4-ae0e-4335-b8c4-cde482ed49e6 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:57.309464 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fa810c4-ae0e-4335-b8c4-cde482ed49e6 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:57.314356 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed852692-5ecf-4165-b364-f5c3a3a7a480 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:57.314959 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed852692-5ecf-4165-b364-f5c3a3a7a480 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:57.315244 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed852692-5ecf-4165-b364-f5c3a3a7a480 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:57.315519 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed852692-5ecf-4165-b364-f5c3a3a7a480 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:57.316169 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed852692-5ecf-4165-b364-f5c3a3a7a480 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:57.317753 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed852692-5ecf-4165-b364-f5c3a3a7a480 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:57.328749 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1449/2903] 10.222.0.90 () {58 vars in 1122 bytes} [Mon Aug 29 21:42:57 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 21:42:57.332881 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4dd9b3f-42c1-47d1-82e0-29589c2170a0 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:57.343335 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4dd9b3f-42c1-47d1-82e0-29589c2170a0 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:57.349136 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9fed206-ddcc-4a31-94bd-bbcd93db993a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:57.349951 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9fed206-ddcc-4a31-94bd-bbcd93db993a admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:57.350241 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9fed206-ddcc-4a31-94bd-bbcd93db993a admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:57.350518 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9fed206-ddcc-4a31-94bd-bbcd93db993a admin admin] PATH_INFO: `/v3/projects/02d9910ba5894e0ea7b8d6fb09a54699/users/78d18520747b44fe8db7aac648ee0a91/roles/02e242d5520740d4b77a5f4008c7eb57` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:57.365559 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9fed206-ddcc-4a31-94bd-bbcd93db993a admin admin] RBAC: Authorizing `identity:create_grant(project_id=02d9910ba5894e0ea7b8d6fb09a54699, user_id=78d18520747b44fe8db7aac648ee0a91, role_id=02e242d5520740d4b77a5f4008c7eb57)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:57.368238 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9fed206-ddcc-4a31-94bd-bbcd93db993a admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:57.381258 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1455/2904] 10.222.0.90 () {60 vars in 1483 bytes} [Mon Aug 29 21:42:57 2022] PUT /identity/v3/projects/02d9910ba5894e0ea7b8d6fb09a54699/users/78d18520747b44fe8db7aac648ee0a91/roles/02e242d5520740d4b77a5f4008c7eb57 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:42:57.393137 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6410be12-4c0b-48ce-ac15-43fea80cab04 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:57.393424 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6410be12-4c0b-48ce-ac15-43fea80cab04 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:57.393700 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6410be12-4c0b-48ce-ac15-43fea80cab04 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:57.407459 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:42:57.424609 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-6410be12-4c0b-48ce-ac15-43fea80cab04 None None] MFA Rules not processed for user `78d18520747b44fe8db7aac648ee0a91`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:42:57.425931 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-6410be12-4c0b-48ce-ac15-43fea80cab04 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:57.491405 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1450/2905] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:42:57 2022] POST /identity/v3/auth/tokens => generated 3274 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:42:57.503298 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9fed206-ddcc-4a31-94bd-bbcd93db993a admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:42:57.505011 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-d9fed206-ddcc-4a31-94bd-bbcd93db993a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:42:57.510500 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:42:57.568196 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9fed206-ddcc-4a31-94bd-bbcd93db993a admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:42:57.571919 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e71b835b-beb1-4c36-974c-9718046601a5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0dc226f7ddcb49ecacb51ce57fdfc11b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:42:57.572536 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e71b835b-beb1-4c36-974c-9718046601a5 service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:42:57.572847 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e71b835b-beb1-4c36-974c-9718046601a5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:42:57.573124 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e71b835b-beb1-4c36-974c-9718046601a5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:42:57.582588 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e71b835b-beb1-4c36-974c-9718046601a5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:42:57.584082 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e71b835b-beb1-4c36-974c-9718046601a5 service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:42:57.594135 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1456/2906] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:42:57 2022] GET /identity/v3/auth/tokens => generated 3274 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:43:07.488583 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=29 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:43:07.589375 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=25 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:43:07.763798 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf9e79b2-1889-4da4-853e-e3eacf0ef137 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:43:07.764155 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf9e79b2-1889-4da4-853e-e3eacf0ef137 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:43:07.764500 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf9e79b2-1889-4da4-853e-e3eacf0ef137 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:43:07.785133 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-bf9e79b2-1889-4da4-853e-e3eacf0ef137 None None] MFA Rules not processed for user `7dceb7dc857348fda46f501288144bde`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:43:07.786373 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-bf9e79b2-1889-4da4-853e-e3eacf0ef137 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:43:07.844369 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1451/2907] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:43:07 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:43:10.951775 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e71b835b-beb1-4c36-974c-9718046601a5 service cinder] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:43:11.032528 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e71b835b-beb1-4c36-974c-9718046601a5 service cinder] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:43:11.037447 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f90cf38-6ef4-477c-b319-38a3131eef58 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:43:11.038136 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f90cf38-6ef4-477c-b319-38a3131eef58 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:43:11.038417 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f90cf38-6ef4-477c-b319-38a3131eef58 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:43:11.038694 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f90cf38-6ef4-477c-b319-38a3131eef58 admin admin] PATH_INFO: `/v3/users/78d18520747b44fe8db7aac648ee0a91` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:43:11.039853 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f90cf38-6ef4-477c-b319-38a3131eef58 admin admin] RBAC: Authorizing `identity:delete_user(user_id=78d18520747b44fe8db7aac648ee0a91)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:43:11.041896 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f90cf38-6ef4-477c-b319-38a3131eef58 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:43:11.098664 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-2f90cf38-6ef4-477c-b319-38a3131eef58 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '78d18520747b44fe8db7aac648ee0a91'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:43:11.113262 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-2f90cf38-6ef4-477c-b319-38a3131eef58 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '78d18520747b44fe8db7aac648ee0a91'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:43:11.114096 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-2f90cf38-6ef4-477c-b319-38a3131eef58 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '78d18520747b44fe8db7aac648ee0a91'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:43:11.114730 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-2f90cf38-6ef4-477c-b319-38a3131eef58 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '78d18520747b44fe8db7aac648ee0a91'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:43:11.122242 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-2f90cf38-6ef4-477c-b319-38a3131eef58 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '78d18520747b44fe8db7aac648ee0a91'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:43:11.151425 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1457/2908] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:43:10 2022] DELETE /identity/v3/users/78d18520747b44fe8db7aac648ee0a91 => generated 0 bytes in 200 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:43:11.156730 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8fa8922-1032-4eaf-a42a-1744c6373e4f None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:43:11.157564 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8fa8922-1032-4eaf-a42a-1744c6373e4f None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:43:11.158065 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8fa8922-1032-4eaf-a42a-1744c6373e4f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:43:11.193165 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-c8fa8922-1032-4eaf-a42a-1744c6373e4f None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:43:11.194469 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-c8fa8922-1032-4eaf-a42a-1744c6373e4f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:43:11.264680 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1452/2909] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:43:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:43:11.275327 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f90cf38-6ef4-477c-b319-38a3131eef58 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:43:11.277095 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-2f90cf38-6ef4-477c-b319-38a3131eef58 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:43:11.349768 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f90cf38-6ef4-477c-b319-38a3131eef58 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:43:11.356709 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaf57893-636f-4a9a-b91a-246a23563989 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:43:11.357337 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaf57893-636f-4a9a-b91a-246a23563989 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:43:11.357626 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaf57893-636f-4a9a-b91a-246a23563989 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:43:11.357942 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaf57893-636f-4a9a-b91a-246a23563989 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:43:11.372170 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaf57893-636f-4a9a-b91a-246a23563989 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:43:11.373479 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaf57893-636f-4a9a-b91a-246a23563989 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:43:11.386300 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1458/2910] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:43:11 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 111 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:43:11.720825 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8fa8922-1032-4eaf-a42a-1744c6373e4f None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:43:11.722652 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-c8fa8922-1032-4eaf-a42a-1744c6373e4f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:43:11.737752 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8fa8922-1032-4eaf-a42a-1744c6373e4f None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:43:11.742969 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36b755e8-8663-413e-a8d8-537921c214f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:43:11.743725 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b755e8-8663-413e-a8d8-537921c214f8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:43:11.744072 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b755e8-8663-413e-a8d8-537921c214f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:43:11.744357 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b755e8-8663-413e-a8d8-537921c214f8 admin admin] PATH_INFO: `/v3/projects/02d9910ba5894e0ea7b8d6fb09a54699` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:43:11.745676 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36b755e8-8663-413e-a8d8-537921c214f8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=02d9910ba5894e0ea7b8d6fb09a54699)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:43:11.748343 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36b755e8-8663-413e-a8d8-537921c214f8 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:43:11.800175 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-36b755e8-8663-413e-a8d8-537921c214f8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '02d9910ba5894e0ea7b8d6fb09a54699', 'request_id': 'req-36b755e8-8663-413e-a8d8-537921c214f8'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:43:11.806137 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-36b755e8-8663-413e-a8d8-537921c214f8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '02d9910ba5894e0ea7b8d6fb09a54699', 'request_id': 'req-36b755e8-8663-413e-a8d8-537921c214f8'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:43:11.814080 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-36b755e8-8663-413e-a8d8-537921c214f8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '02d9910ba5894e0ea7b8d6fb09a54699', 'request_id': 'req-36b755e8-8663-413e-a8d8-537921c214f8'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:43:11.829067 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-36b755e8-8663-413e-a8d8-537921c214f8 admin admin] The token cache is being invalidate because project 02d9910ba5894e0ea7b8d6fb09a54699 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81434) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:43:11.829266 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-36b755e8-8663-413e-a8d8-537921c214f8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:43:11.829918 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-36b755e8-8663-413e-a8d8-537921c214f8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:43:11.831466 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1453/2911] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:43:11 2022] DELETE /identity/v3/projects/02d9910ba5894e0ea7b8d6fb09a54699 => generated 0 bytes in 111 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 21:43:15.568333 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaf57893-636f-4a9a-b91a-246a23563989 service neutron] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:43:15.569827 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-aaf57893-636f-4a9a-b91a-246a23563989 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:43:15.583377 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaf57893-636f-4a9a-b91a-246a23563989 service neutron] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:43:15.601965 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:43:15.646417 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7116a621-d99e-41e4-bbdf-10ac41bba04b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:43:15.647054 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7116a621-d99e-41e4-bbdf-10ac41bba04b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:43:15.647330 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7116a621-d99e-41e4-bbdf-10ac41bba04b admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:43:15.647604 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7116a621-d99e-41e4-bbdf-10ac41bba04b admin admin] PATH_INFO: `/v3/users/a82ca052e3bd4b898bf5623f4e1ed711` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:43:15.648718 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7116a621-d99e-41e4-bbdf-10ac41bba04b admin admin] RBAC: Authorizing `identity:delete_user(user_id=a82ca052e3bd4b898bf5623f4e1ed711)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:43:15.650739 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7116a621-d99e-41e4-bbdf-10ac41bba04b admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:43:15.697538 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-7116a621-d99e-41e4-bbdf-10ac41bba04b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a82ca052e3bd4b898bf5623f4e1ed711'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:43:15.711128 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-7116a621-d99e-41e4-bbdf-10ac41bba04b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a82ca052e3bd4b898bf5623f4e1ed711'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:43:15.712561 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-7116a621-d99e-41e4-bbdf-10ac41bba04b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a82ca052e3bd4b898bf5623f4e1ed711'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:43:15.713250 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-7116a621-d99e-41e4-bbdf-10ac41bba04b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a82ca052e3bd4b898bf5623f4e1ed711'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:43:15.722458 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-7116a621-d99e-41e4-bbdf-10ac41bba04b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a82ca052e3bd4b898bf5623f4e1ed711'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:43:15.737479 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1459/2912] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:43:15 2022] DELETE /identity/v3/users/a82ca052e3bd4b898bf5623f4e1ed711 => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:43:15.743068 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b21cac7-8513-4f05-bed3-b797e6721a79 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:43:15.743582 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b21cac7-8513-4f05-bed3-b797e6721a79 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:43:15.743695 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b21cac7-8513-4f05-bed3-b797e6721a79 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:43:15.768587 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-8b21cac7-8513-4f05-bed3-b797e6721a79 None None] MFA Rules not processed for user `97a0b159abf44cd59fe5a2b7bde28adf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:43:15.769759 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-8b21cac7-8513-4f05-bed3-b797e6721a79 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:43:15.795211 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:43:15.836465 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1454/2913] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:43:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:43:15.848321 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7116a621-d99e-41e4-bbdf-10ac41bba04b admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:43:15.849991 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-7116a621-d99e-41e4-bbdf-10ac41bba04b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:43:15.920249 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7116a621-d99e-41e4-bbdf-10ac41bba04b admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:43:15.925273 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-033eb242-4859-4fec-89fe-4de258d68a50 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:43:15.925949 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-033eb242-4859-4fec-89fe-4de258d68a50 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:43:15.926198 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-033eb242-4859-4fec-89fe-4de258d68a50 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:43:15.926467 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-033eb242-4859-4fec-89fe-4de258d68a50 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:43:15.936959 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-033eb242-4859-4fec-89fe-4de258d68a50 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:43:15.938738 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-033eb242-4859-4fec-89fe-4de258d68a50 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:43:15.948701 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1460/2914] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:43:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:43:16.231351 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b21cac7-8513-4f05-bed3-b797e6721a79 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:43:16.232868 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-8b21cac7-8513-4f05-bed3-b797e6721a79 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:43:16.248457 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b21cac7-8513-4f05-bed3-b797e6721a79 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:43:16.254069 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-758eccc1-814c-43fb-b02a-b162c1685d59 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:43:16.255316 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-758eccc1-814c-43fb-b02a-b162c1685d59 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:43:16.255786 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-758eccc1-814c-43fb-b02a-b162c1685d59 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:43:16.256073 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-758eccc1-814c-43fb-b02a-b162c1685d59 admin admin] PATH_INFO: `/v3/projects/9267783151d64683997ea5302e2e6ec1` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:43:16.257236 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-758eccc1-814c-43fb-b02a-b162c1685d59 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9267783151d64683997ea5302e2e6ec1)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:43:16.259173 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-758eccc1-814c-43fb-b02a-b162c1685d59 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:43:16.308370 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-758eccc1-814c-43fb-b02a-b162c1685d59 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9267783151d64683997ea5302e2e6ec1', 'request_id': 'req-758eccc1-814c-43fb-b02a-b162c1685d59'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:43:16.314194 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-758eccc1-814c-43fb-b02a-b162c1685d59 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9267783151d64683997ea5302e2e6ec1', 'request_id': 'req-758eccc1-814c-43fb-b02a-b162c1685d59'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:43:16.320895 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-758eccc1-814c-43fb-b02a-b162c1685d59 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9267783151d64683997ea5302e2e6ec1', 'request_id': 'req-758eccc1-814c-43fb-b02a-b162c1685d59'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:43:16.334420 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-758eccc1-814c-43fb-b02a-b162c1685d59 admin admin] The token cache is being invalidate because project 9267783151d64683997ea5302e2e6ec1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81434) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:43:16.334714 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-758eccc1-814c-43fb-b02a-b162c1685d59 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:43:16.335347 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-758eccc1-814c-43fb-b02a-b162c1685d59 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:43:16.336918 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1455/2915] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:43:16 2022] DELETE /identity/v3/projects/9267783151d64683997ea5302e2e6ec1 => generated 0 bytes in 106 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:43:23.194141 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-033eb242-4859-4fec-89fe-4de258d68a50 service neutron] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:43:23.195601 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-033eb242-4859-4fec-89fe-4de258d68a50 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:43:23.207873 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-033eb242-4859-4fec-89fe-4de258d68a50 service neutron] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:43:23.254869 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:43:23.265077 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84c51222-0315-491b-a753-f52a46b724ed service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:43:23.267050 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84c51222-0315-491b-a753-f52a46b724ed service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:43:23.267546 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84c51222-0315-491b-a753-f52a46b724ed service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:43:23.268008 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84c51222-0315-491b-a753-f52a46b724ed service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:43:23.269883 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-84c51222-0315-491b-a753-f52a46b724ed service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:43:23.346702 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84c51222-0315-491b-a753-f52a46b724ed service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:43:23.348191 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84c51222-0315-491b-a753-f52a46b724ed service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:43:23.357974 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1461/2916] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:43:23 2022] GET /identity/v3/auth/tokens => generated 3319 bytes in 164 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:43:26.331288 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=41,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:43:33.353546 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=30 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:44:39.023224 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-758eccc1-814c-43fb-b02a-b162c1685d59 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:44:39.024647 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-758eccc1-814c-43fb-b02a-b162c1685d59 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:44:39.037107 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-758eccc1-814c-43fb-b02a-b162c1685d59 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:44:39.040671 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbfb0669-873b-481c-adcb-92988ac2f6c8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:44:39.041270 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbfb0669-873b-481c-adcb-92988ac2f6c8 service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:44:39.041546 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbfb0669-873b-481c-adcb-92988ac2f6c8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:44:39.041847 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbfb0669-873b-481c-adcb-92988ac2f6c8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:44:39.043523 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-fbfb0669-873b-481c-adcb-92988ac2f6c8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:44:39.053355 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbfb0669-873b-481c-adcb-92988ac2f6c8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:44:39.054817 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbfb0669-873b-481c-adcb-92988ac2f6c8 service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:44:39.062842 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1456/2917] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:44:39 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:44:49.059220 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=6 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:45:17.820331 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84c51222-0315-491b-a753-f52a46b724ed service neutron] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:45:17.829340 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84c51222-0315-491b-a753-f52a46b724ed service neutron] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:45:17.833498 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-612a4607-fdb1-4dbe-a78b-dd46419b8542 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:45:17.834120 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-612a4607-fdb1-4dbe-a78b-dd46419b8542 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:45:17.834395 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-612a4607-fdb1-4dbe-a78b-dd46419b8542 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:45:17.834664 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-612a4607-fdb1-4dbe-a78b-dd46419b8542 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:45:17.836373 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-612a4607-fdb1-4dbe-a78b-dd46419b8542 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:45:17.912775 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-612a4607-fdb1-4dbe-a78b-dd46419b8542 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:45:17.914243 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-612a4607-fdb1-4dbe-a78b-dd46419b8542 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:45:17.925998 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1462/2918] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:45:17 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:45:19.115423 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbfb0669-873b-481c-adcb-92988ac2f6c8 service neutron] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:45:19.124071 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbfb0669-873b-481c-adcb-92988ac2f6c8 service neutron] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:45:19.127675 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2da299ee-d7bf-435f-9d2a-eacf48dcf8f4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:45:19.128451 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2da299ee-d7bf-435f-9d2a-eacf48dcf8f4 service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:45:19.128741 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2da299ee-d7bf-435f-9d2a-eacf48dcf8f4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:45:19.129021 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2da299ee-d7bf-435f-9d2a-eacf48dcf8f4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:45:19.130786 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-2da299ee-d7bf-435f-9d2a-eacf48dcf8f4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:45:19.144645 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2da299ee-d7bf-435f-9d2a-eacf48dcf8f4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:45:19.146077 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2da299ee-d7bf-435f-9d2a-eacf48dcf8f4 service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:45:19.157459 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1457/2919] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:45:19 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:45:20.258901 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-612a4607-fdb1-4dbe-a78b-dd46419b8542 service neutron] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:45:20.260911 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-612a4607-fdb1-4dbe-a78b-dd46419b8542 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:45:20.275981 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-612a4607-fdb1-4dbe-a78b-dd46419b8542 service neutron] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:45:20.282146 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94af4b2f-a2da-4877-b924-1a5d0396f2a3 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a35b5b7dea7b421da056f063d2436ffb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:45:20.282791 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94af4b2f-a2da-4877-b924-1a5d0396f2a3 service nova] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:45:20.283075 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94af4b2f-a2da-4877-b924-1a5d0396f2a3 service nova] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:45:20.283415 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94af4b2f-a2da-4877-b924-1a5d0396f2a3 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:45:20.299358 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-94af4b2f-a2da-4877-b924-1a5d0396f2a3 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:45:20.344888 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94af4b2f-a2da-4877-b924-1a5d0396f2a3 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:45:20.350843 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94af4b2f-a2da-4877-b924-1a5d0396f2a3 service nova] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:45:20.373455 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1463/2920] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:45:20 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 115 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:45:21.457156 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf5961aa-93d0-4dd8-85e1-0be8252de011 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:45:21.457738 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf5961aa-93d0-4dd8-85e1-0be8252de011 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:45:21.457738 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf5961aa-93d0-4dd8-85e1-0be8252de011 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:45:21.479810 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-cf5961aa-93d0-4dd8-85e1-0be8252de011 None None] MFA Rules not processed for user `7dceb7dc857348fda46f501288144bde`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:45:21.481060 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-cf5961aa-93d0-4dd8-85e1-0be8252de011 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:45:21.556406 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1458/2921] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:45:21 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:45:21.887248 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94af4b2f-a2da-4877-b924-1a5d0396f2a3 service nova] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:45:21.897065 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94af4b2f-a2da-4877-b924-1a5d0396f2a3 service nova] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:45:21.900662 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-045b0381-59a9-42dc-a394-31736e73df33 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:45:21.901267 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-045b0381-59a9-42dc-a394-31736e73df33 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:45:21.901550 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-045b0381-59a9-42dc-a394-31736e73df33 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:45:21.901858 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-045b0381-59a9-42dc-a394-31736e73df33 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:45:21.903603 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-045b0381-59a9-42dc-a394-31736e73df33 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:45:21.914553 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-045b0381-59a9-42dc-a394-31736e73df33 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:45:21.916016 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-045b0381-59a9-42dc-a394-31736e73df33 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:45:21.925049 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1464/2922] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:45:21 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:45:22.434249 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf5961aa-93d0-4dd8-85e1-0be8252de011 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:45:22.435760 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-cf5961aa-93d0-4dd8-85e1-0be8252de011 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:45:22.517557 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf5961aa-93d0-4dd8-85e1-0be8252de011 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:45:22.523975 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17e14a68-9378-4529-89a5-e01c42b81614 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b95fa07a7c0549dbaac49406e2639e21', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:45:22.524599 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17e14a68-9378-4529-89a5-e01c42b81614 service placement] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:45:22.524909 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17e14a68-9378-4529-89a5-e01c42b81614 service placement] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:45:22.525183 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17e14a68-9378-4529-89a5-e01c42b81614 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:45:22.527954 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-17e14a68-9378-4529-89a5-e01c42b81614 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:45:22.540541 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17e14a68-9378-4529-89a5-e01c42b81614 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:45:22.542178 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17e14a68-9378-4529-89a5-e01c42b81614 service placement] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:45:22.552221 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1459/2923] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:45:22 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 119 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:45:31.920807 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=46 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:45:32.547796 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=63 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:45:34.115991 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-045b0381-59a9-42dc-a394-31736e73df33 service neutron] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:45:34.130858 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-045b0381-59a9-42dc-a394-31736e73df33 service neutron] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:45:34.136911 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98b260b8-7013-4e94-8c86-c7060fe3b488 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a35b5b7dea7b421da056f063d2436ffb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:45:34.137533 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98b260b8-7013-4e94-8c86-c7060fe3b488 service nova] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:45:34.137852 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98b260b8-7013-4e94-8c86-c7060fe3b488 service nova] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:45:34.138135 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98b260b8-7013-4e94-8c86-c7060fe3b488 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:45:34.139902 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-98b260b8-7013-4e94-8c86-c7060fe3b488 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:45:34.154670 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98b260b8-7013-4e94-8c86-c7060fe3b488 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:45:34.156693 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98b260b8-7013-4e94-8c86-c7060fe3b488 service nova] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:45:34.168613 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1465/2924] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:45:34 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 54 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:45:44.163964 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=6 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:46:41.694036 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17e14a68-9378-4529-89a5-e01c42b81614 service placement] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:46:41.695467 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-17e14a68-9378-4529-89a5-e01c42b81614 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:46:41.709477 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17e14a68-9378-4529-89a5-e01c42b81614 service placement] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:46:41.714814 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-048354c8-6a1e-4bb7-9e21-6b5e6c2d97d4 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b95fa07a7c0549dbaac49406e2639e21', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:46:41.715422 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-048354c8-6a1e-4bb7-9e21-6b5e6c2d97d4 service placement] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:46:41.715705 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-048354c8-6a1e-4bb7-9e21-6b5e6c2d97d4 service placement] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:46:41.715979 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-048354c8-6a1e-4bb7-9e21-6b5e6c2d97d4 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:46:41.717776 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-048354c8-6a1e-4bb7-9e21-6b5e6c2d97d4 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:46:41.730694 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-048354c8-6a1e-4bb7-9e21-6b5e6c2d97d4 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:46:41.732266 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-048354c8-6a1e-4bb7-9e21-6b5e6c2d97d4 service placement] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:46:41.743768 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1460/2925] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:46:41 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 50 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:46:51.739250 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=6 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:47:11.717386 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4c86bed-e86d-4bbb-b88f-8118c1257586 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:47:11.717960 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4c86bed-e86d-4bbb-b88f-8118c1257586 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:47:11.717960 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4c86bed-e86d-4bbb-b88f-8118c1257586 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:47:11.737743 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-c4c86bed-e86d-4bbb-b88f-8118c1257586 None None] MFA Rules not processed for user `7dceb7dc857348fda46f501288144bde`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 21:47:11.738966 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-c4c86bed-e86d-4bbb-b88f-8118c1257586 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:47:11.746324 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1466/2926] 10.222.0.90 () {58 vars in 958 bytes} [Mon Aug 29 21:47:11 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 21:47:12.079777 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-048354c8-6a1e-4bb7-9e21-6b5e6c2d97d4 service placement] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:47:12.089929 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-048354c8-6a1e-4bb7-9e21-6b5e6c2d97d4 service placement] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:47:12.093545 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b59e58d7-9f39-4224-b738-fbb3f2accddc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:47:12.094255 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b59e58d7-9f39-4224-b738-fbb3f2accddc service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:47:12.094537 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b59e58d7-9f39-4224-b738-fbb3f2accddc service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:47:12.094811 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b59e58d7-9f39-4224-b738-fbb3f2accddc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:47:12.097104 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-b59e58d7-9f39-4224-b738-fbb3f2accddc service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:47:12.107797 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b59e58d7-9f39-4224-b738-fbb3f2accddc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:47:12.109340 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b59e58d7-9f39-4224-b738-fbb3f2accddc service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:47:12.118841 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1461/2927] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:47:12 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:47:21.731762 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:47:22.114713 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=6 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:48:56.580655 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4c86bed-e86d-4bbb-b88f-8118c1257586 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:48:56.590384 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4c86bed-e86d-4bbb-b88f-8118c1257586 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:48:56.593916 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a10259bd-62b5-4aee-abd9-37f49fd8bca6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:48:56.594563 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a10259bd-62b5-4aee-abd9-37f49fd8bca6 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:48:56.594847 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a10259bd-62b5-4aee-abd9-37f49fd8bca6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:48:56.595117 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a10259bd-62b5-4aee-abd9-37f49fd8bca6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:48:56.597092 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-a10259bd-62b5-4aee-abd9-37f49fd8bca6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:48:56.608948 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a10259bd-62b5-4aee-abd9-37f49fd8bca6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:48:56.611693 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a10259bd-62b5-4aee-abd9-37f49fd8bca6 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:48:56.620964 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1467/2928] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:48:56 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:49:06.616772 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=6 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:49:10.282847 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b59e58d7-9f39-4224-b738-fbb3f2accddc service neutron] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:49:10.291607 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b59e58d7-9f39-4224-b738-fbb3f2accddc service neutron] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:49:10.295231 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06a614f3-0771-4713-bfca-25df3e541148 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e69b7f0c3574abc9de5fb580d22152a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:49:10.295860 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06a614f3-0771-4713-bfca-25df3e541148 service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:49:10.296139 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06a614f3-0771-4713-bfca-25df3e541148 service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:49:10.296414 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06a614f3-0771-4713-bfca-25df3e541148 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:49:10.298779 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-06a614f3-0771-4713-bfca-25df3e541148 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:49:10.369455 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06a614f3-0771-4713-bfca-25df3e541148 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:49:10.370921 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06a614f3-0771-4713-bfca-25df3e541148 service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:49:10.380693 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1462/2929] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:49:10 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:49:13.503360 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a10259bd-62b5-4aee-abd9-37f49fd8bca6 service neutron] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:49:13.517643 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a10259bd-62b5-4aee-abd9-37f49fd8bca6 service neutron] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:49:13.524124 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93a26bc0-51f2-4cda-b1a6-5a68be2ea3e5 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b95fa07a7c0549dbaac49406e2639e21', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:49:13.525152 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93a26bc0-51f2-4cda-b1a6-5a68be2ea3e5 service placement] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:49:13.525539 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93a26bc0-51f2-4cda-b1a6-5a68be2ea3e5 service placement] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:49:13.525962 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93a26bc0-51f2-4cda-b1a6-5a68be2ea3e5 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:49:13.527981 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-93a26bc0-51f2-4cda-b1a6-5a68be2ea3e5 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:49:13.542750 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93a26bc0-51f2-4cda-b1a6-5a68be2ea3e5 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:49:13.544844 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93a26bc0-51f2-4cda-b1a6-5a68be2ea3e5 service placement] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:49:13.556206 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1468/2930] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:49:13 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 53 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:49:16.776274 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06a614f3-0771-4713-bfca-25df3e541148 service neutron] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:49:16.777627 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-06a614f3-0771-4713-bfca-25df3e541148 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:49:16.795056 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06a614f3-0771-4713-bfca-25df3e541148 service neutron] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:49:16.801728 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eacedb3-dc8b-4a0d-a592-d74f70011cbe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:49:16.802447 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eacedb3-dc8b-4a0d-a592-d74f70011cbe admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:49:16.802797 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eacedb3-dc8b-4a0d-a592-d74f70011cbe admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:49:16.803083 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eacedb3-dc8b-4a0d-a592-d74f70011cbe admin admin] PATH_INFO: `/v3/users/7dceb7dc857348fda46f501288144bde` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:49:16.804274 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eacedb3-dc8b-4a0d-a592-d74f70011cbe admin admin] RBAC: Authorizing `identity:delete_user(user_id=7dceb7dc857348fda46f501288144bde)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:49:16.806268 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eacedb3-dc8b-4a0d-a592-d74f70011cbe admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:49:16.873828 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-1eacedb3-dc8b-4a0d-a592-d74f70011cbe admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7dceb7dc857348fda46f501288144bde'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:49:16.882520 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-1eacedb3-dc8b-4a0d-a592-d74f70011cbe admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7dceb7dc857348fda46f501288144bde'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:49:16.890490 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-1eacedb3-dc8b-4a0d-a592-d74f70011cbe admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7dceb7dc857348fda46f501288144bde'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:49:16.891842 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-1eacedb3-dc8b-4a0d-a592-d74f70011cbe admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7dceb7dc857348fda46f501288144bde'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:49:16.892490 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-1eacedb3-dc8b-4a0d-a592-d74f70011cbe admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7dceb7dc857348fda46f501288144bde'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:49:16.911242 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1463/2931] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:49:16 2022] DELETE /identity/v3/users/7dceb7dc857348fda46f501288144bde => generated 0 bytes in 136 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:49:17.085311 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93a26bc0-51f2-4cda-b1a6-5a68be2ea3e5 service placement] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:49:17.097687 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93a26bc0-51f2-4cda-b1a6-5a68be2ea3e5 service placement] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:49:17.157939 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-816cbc1a-f030-44b5-9c49-5e155bde9946 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:49:17.158576 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-816cbc1a-f030-44b5-9c49-5e155bde9946 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:49:17.158860 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-816cbc1a-f030-44b5-9c49-5e155bde9946 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:49:17.159138 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-816cbc1a-f030-44b5-9c49-5e155bde9946 admin admin] PATH_INFO: `/v3/projects/1ba68826bbc24418aabf0c66294578ca` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:49:17.160267 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-816cbc1a-f030-44b5-9c49-5e155bde9946 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1ba68826bbc24418aabf0c66294578ca)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:49:17.162454 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-816cbc1a-f030-44b5-9c49-5e155bde9946 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:49:17.209103 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-816cbc1a-f030-44b5-9c49-5e155bde9946 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1ba68826bbc24418aabf0c66294578ca', 'request_id': 'req-816cbc1a-f030-44b5-9c49-5e155bde9946'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:49:17.215417 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-816cbc1a-f030-44b5-9c49-5e155bde9946 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1ba68826bbc24418aabf0c66294578ca', 'request_id': 'req-816cbc1a-f030-44b5-9c49-5e155bde9946'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:49:17.221713 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-816cbc1a-f030-44b5-9c49-5e155bde9946 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1ba68826bbc24418aabf0c66294578ca', 'request_id': 'req-816cbc1a-f030-44b5-9c49-5e155bde9946'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:49:17.233444 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-816cbc1a-f030-44b5-9c49-5e155bde9946 admin admin] The token cache is being invalidate because project 1ba68826bbc24418aabf0c66294578ca was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:49:17.233730 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-816cbc1a-f030-44b5-9c49-5e155bde9946 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:49:17.234367 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-816cbc1a-f030-44b5-9c49-5e155bde9946 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:49:17.236433 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1469/2932] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:49:17 2022] DELETE /identity/v3/projects/1ba68826bbc24418aabf0c66294578ca => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:49:17.240438 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eacedb3-dc8b-4a0d-a592-d74f70011cbe admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:49:17.241874 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-1eacedb3-dc8b-4a0d-a592-d74f70011cbe admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:49:17.254833 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eacedb3-dc8b-4a0d-a592-d74f70011cbe admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:49:17.309940 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-343dc734-37da-4816-a9af-768024f4adeb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:49:17.310569 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-343dc734-37da-4816-a9af-768024f4adeb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:49:17.310848 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-343dc734-37da-4816-a9af-768024f4adeb admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:49:17.311116 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-343dc734-37da-4816-a9af-768024f4adeb admin admin] PATH_INFO: `/v3/users/f20085430c7e4d5983c4880fe8bc2b58` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:49:17.312239 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-343dc734-37da-4816-a9af-768024f4adeb admin admin] RBAC: Authorizing `identity:delete_user(user_id=f20085430c7e4d5983c4880fe8bc2b58)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:49:17.314276 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-343dc734-37da-4816-a9af-768024f4adeb admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:49:17.319542 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:49:17.362277 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-343dc734-37da-4816-a9af-768024f4adeb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f20085430c7e4d5983c4880fe8bc2b58'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:49:17.369146 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-343dc734-37da-4816-a9af-768024f4adeb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f20085430c7e4d5983c4880fe8bc2b58'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:49:17.376875 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-343dc734-37da-4816-a9af-768024f4adeb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f20085430c7e4d5983c4880fe8bc2b58'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:49:17.377952 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-343dc734-37da-4816-a9af-768024f4adeb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f20085430c7e4d5983c4880fe8bc2b58'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:49:17.378697 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-343dc734-37da-4816-a9af-768024f4adeb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f20085430c7e4d5983c4880fe8bc2b58'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:49:17.397871 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1464/2933] 10.222.0.90 () {58 vars in 1224 bytes} [Mon Aug 29 21:49:17 2022] DELETE /identity/v3/users/f20085430c7e4d5983c4880fe8bc2b58 => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:49:17.577465 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-816cbc1a-f030-44b5-9c49-5e155bde9946 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:49:17.579871 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-816cbc1a-f030-44b5-9c49-5e155bde9946 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:49:17.594883 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-816cbc1a-f030-44b5-9c49-5e155bde9946 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:49:17.655952 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32b21751-d7f6-4bf4-8e5b-557354ee1aba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '97a0b159abf44cd59fe5a2b7bde28adf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa1ebaa0d49241599f614eb2efe4d8c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:49:17.656749 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32b21751-d7f6-4bf4-8e5b-557354ee1aba admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:49:17.657234 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32b21751-d7f6-4bf4-8e5b-557354ee1aba admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:49:17.657518 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32b21751-d7f6-4bf4-8e5b-557354ee1aba admin admin] PATH_INFO: `/v3/projects/c0cedb3be4f24389b32089be9afb03b1` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:49:17.658802 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32b21751-d7f6-4bf4-8e5b-557354ee1aba admin admin] RBAC: Authorizing `identity:delete_project(project_id=c0cedb3be4f24389b32089be9afb03b1)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:49:17.660740 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32b21751-d7f6-4bf4-8e5b-557354ee1aba admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:49:17.681293 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:49:17.709738 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-32b21751-d7f6-4bf4-8e5b-557354ee1aba admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c0cedb3be4f24389b32089be9afb03b1', 'request_id': 'req-32b21751-d7f6-4bf4-8e5b-557354ee1aba'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:49:17.715983 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-32b21751-d7f6-4bf4-8e5b-557354ee1aba admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c0cedb3be4f24389b32089be9afb03b1', 'request_id': 'req-32b21751-d7f6-4bf4-8e5b-557354ee1aba'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:49:17.721713 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-32b21751-d7f6-4bf4-8e5b-557354ee1aba admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c0cedb3be4f24389b32089be9afb03b1', 'request_id': 'req-32b21751-d7f6-4bf4-8e5b-557354ee1aba'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:49:17.735911 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-32b21751-d7f6-4bf4-8e5b-557354ee1aba admin admin] The token cache is being invalidate because project c0cedb3be4f24389b32089be9afb03b1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 21:49:17.736204 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-32b21751-d7f6-4bf4-8e5b-557354ee1aba admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:49:17.736834 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-32b21751-d7f6-4bf4-8e5b-557354ee1aba admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 21:49:17.738427 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1470/2934] 10.222.0.90 () {58 vars in 1233 bytes} [Mon Aug 29 21:49:17 2022] DELETE /identity/v3/projects/c0cedb3be4f24389b32089be9afb03b1 => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 21:49:27.394258 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=23,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:49:27.732956 dv-cinder-854267-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=16,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 21:51:43.707684 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-343dc734-37da-4816-a9af-768024f4adeb admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 21:51:43.709138 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-343dc734-37da-4816-a9af-768024f4adeb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:51:43.854106 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-343dc734-37da-4816-a9af-768024f4adeb admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 21:51:43.916827 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97cff470-d0a0-4261-87c2-2b294993711d service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b95fa07a7c0549dbaac49406e2639e21', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ec03e112e3d547fbab6faf7b485c3fca', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 21:51:43.917456 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97cff470-d0a0-4261-87c2-2b294993711d service placement] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 21:51:43.917771 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97cff470-d0a0-4261-87c2-2b294993711d service placement] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 21:51:43.918091 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97cff470-d0a0-4261-87c2-2b294993711d service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 21:51:43.919888 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-97cff470-d0a0-4261-87c2-2b294993711d service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 21:51:43.932048 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97cff470-d0a0-4261-87c2-2b294993711d service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 21:51:43.933628 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97cff470-d0a0-4261-87c2-2b294993711d service placement] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 21:51:43.943929 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1465/2935] 10.222.0.90 () {60 vars in 1365 bytes} [Mon Aug 29 21:51:43 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 237 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 21:51:53.940162 dv-cinder-854267-3-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=89 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}